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We consider the problem of scheduling a queueing system in 
which many statistically identical servers cater to several classes of 
impatient customers. Service times and impatience clocks are expo- 
nential while arrival processes are renewal. Our cost is an expected 
cumulative discounted function, linear or nonlinear, of appropriately 
normalized performance measures. As a special case, the cost per 
unit time can be a function of the number of customers waiting to be 
served in each class, the number actually being served, the abandon- 
ment rate, the delay experienced by customers, the number of idling 
servers, as well as certain combinations thereof. We study the system 
in an asymptotic heavy-traffic regime where the number of servers 
n and the offered load r are simultaneously scaled up and carefully 
balanced: n « r + /3\/r for some scalar /3. This yields an operation 
that enjoys the benefits of both heavy traffic (high server utilization) 
and light traffic (high service levels.) 

We first consider a formal weak limit, through which our queue- 
ing scheduling problem gives rise to a diffusion control problem. We 
show that the latter has an optimal Markov control policy, and that 
the corresponding Hamilton-Jacobi-Bellman (HJB) equation has a 
unique classical solution. The Markov control policy and the HJB 
equation are then used to define scheduling control policies which 
we prove are asymptotically optimal for our original queueing sys- 
tem. The analysis yields both qualitative and quantitative insights. 
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in particular on staffing levels, the roles of non-preemption and work 
conservation, and the trade-off between service quality and servers' 
efHciency. 
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1. Introduction. We analyze a queueing system that consists of sev- 
eral customer classes and a large pool of independent statistically identi- 
cal servers (see Figure 1). Customer arrivals for each class follow a renewal 
process. Each server can serve customers of all classes, and service dura- 
tions are exponentially distributed with class-dependent means. In addition, 
some customers abandon the system while waiting to be served, and aban- 
donments arise according to exponential clocks with class-dependent rates. 
This work addresses the stochastic control problem of system scheduling: 
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Fig. 1. A many-server multiclass queueing system. 

how to optimally match customers and servers. The cost criterion we con- 
sider is an expected cumulative discounted function of the (appropriately 
normalized) number of customers waiting to be served and the number ac- 
tually being served, for each class. Special cases for the cost per unit time 
are the number of customers in the system (or increasing functions of it), 
the number of abandonments per unit time, the delay experienced by the 
customers, the number of idling servers and certain combinations of these 
costs. Since our scheduling problem is too complex for direct analysis, we 
resort to heavy-traffic asymptotics. The goal is to identify the asymptotics 
with a diffusion control problem, then rigorously justify this identification 
and finally gain insight from it. 

1.1. Motivation: the QED regime. The asymptotic heavy-traffic regime 
that we consider is the one analyzed by Jagerman [24], Halfin and Whitt [17] 
and Fleming, Stolyar and Simon [12]. Here, the number of servers and the 
arrival rates are large and carefully balanced so that the traffic intensity is 
moderately close to unity. Economies of scale then enable an operation that 
is both efficiency-driven (high servers' utilization) and quality-driven (high 
service levels), hence the terminology QED: both Quality- and Efficiency- 
driven. 

An important motivating application for our model is the modern tele- 
phone call center, where a large heterogeneous customer population seeks 
service from many flexible servers. In this context, the QED regime was 
identified in practice first by Sze [36], and more recently and systematically 
in Garnett, Mandelbaum and Reiman [15]. The QED regime captures the 
operational environment of well-run moderate-to-large call centers, where 
servers' utilization is high yet a significant fraction of the customers is 
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served immediately upon calling. The last two statements are in fact equiv- 
alent for single-class many-server systems [15, 17]. They are further equiva- 
lent to "square-root safety staffing," which also applies to the model under 
study here: if r denotes the offered load and n the number of servers, then 
n ~ r + Py/r for some constant /3. (See [13] for more elaboration, motivation 
and references.) 

For a single-class queue {GI/M/n) in the QED regime, one subtracts 
from the number of customers in the system the number of servers and then 
divides by the square root of the latter. The resulting stochastic process, 
when positive, models the (scaled) queue-length, and when negative models 
the (scaled) number of idle servers. Halfin and Whitt [17] proved that this 
process converges in distribution, as the number of servers (n) grows without 
bound, to a diffusion process with a fixed diffusion coefficient and a piece- 
wise linear state-dependent drift, under appropriate assumptions on system 
parameters. The result was extended in [15] to accommodate abandonment 
from the queue (but arrivals were assumed Poisson). Further extensions were 
carried out by Puhalskii and Reiman [34] to cover a multiclass queue, phase- 
type service time distributions and priority scheduling policies, giving rise 
in the limit to a multidimensional diffusion process. 

1.2. Diffusion control problems and queueing systems. There has been a 
considerable amount of research on diffusion control problems in the context 
of queueing systems, specifically on asymptotic optimality when approach- 
ing a diffusive limit. We refer the reader to [38] for a summary and further 
references. Most of this research, however, has been within the "conven- 
tional" heavy-traffic regime which, in the terminology introduced above, 
corresponds to an efficiency-driven regime of operation: servers' utilization 
approaches 100%, with essentially all customers being delayed in queue for 
service. To wit, our model in "conventional" heavy traffic was analyzed by 
Van Mieghem [37], who considered a single server (or equivalently, a fixed 
number of servers) with traffic intensity converging to unity. (One could, 
alternatively, increase the number of servers to infinity, which entails an 
acceleration of the convergence to unity; see the last section of [30].) 

Following Harrison [18], there has been a stream of research that produced 
schemes for determining "good" scheduling policies for queueing systems, in 
an asymptotic sense. These have been based on exact analytic solutions 
to corresponding diffusion control problems, formally obtained as "conven- 
tional" heavy traffic limits. For rigorous proofs of asymptotic optimality, see 
[[3, 26, 27, 29, 30, 31, 32] and [37]]. 

Recently, Armony and Maglaras [1], Harrison and Zeevi [21] and the 
present authors [2] have considered stochastic control problems in the QED 
regime. The first [1] models and analyzes rational customers in equilibrium, 
and the last [2] served as a pilot for the present paper. The analysis in [21] 
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is that of the diffusion control problem associated with our queueing sys- 
tem with linear costs. Specifically, Harrison and Zeevi show in [21] that this 
control problem has an optimal Markov control policy (cf. [11]) which is 
characterized in terms of its underlying HJB equation. Then, they use the 
diffusion control problem to propose a scheduling control policy for the orig- 
inal queueing system, conjecturing that it is asymptotically optimal in the 
QED regime. In the current paper we use that same approach, with yet a 
significant broadening of modeling scope: we identify a sequence of HJB- 
based scheduling policies (for a general and natural cost structure) and we 
prove their asymptotic optimality (within a broad family of nonanticipating 
preemptive or nonpreemptive policies). 

1.3. Main results and scope. Our main results are as follows. First, we 
formally take a heavy-traffic limit in the QED regime (Section 2.3). Then 
we show that the diffusion control problem associated with this limit has 
an optimal Markov control policy, and that its HJB equation has a unique 
classical solution (see Theorem 3). This extends the results of [21] to cover 
a large class of cost functions. As is often the case in stochastic control of 
diffusions, proving existence of optimal Markov control policies is coupled 
with establishing the existence and uniqueness of solutions for the underlying 
HJB equation. In the case of bounded cost, existence and uniqueness for this 
equation follow from the theory of optimal control of diffusions [6, 11] and 
of nonlinear elliptic PDEs [23] . Since our cost is not assumed to be bounded, 
finer information on the model needs to be exploited, and in particular mo- 
ment estimates on the controlled processes are required [Proposition 4(ii)]. 

Having studied the diffusion control problem and the HJB equation, we 
use them to propose a scheme for determining scheduling control policies of 
two types: preemptive and nonpreemptive (see Section 2.6). After defining 
a notion of scheduling control policies that do not anticipate the future, we 
prove that among them, our proposed policies are asymptotically optimal 
in the QED heavy-traffic limit (Theorems 2 and 4). (More precisely, asymp- 
totic optimality is proved among work conserving policies; more on that in 
the sequel.) The asymptotic optimality is in the sense that, under the pro- 
posed policies, the cost converges to the optimal cost of the diffusion control 
problem, and that the latter is a lower bound for the limit inferior of costs 
under any other sequence of policies. 

Our approach for deriving the diffusion control problem follows Bell and 
Williams [3] in that the system of equations and the cost are represented 
in terms of the system's primitives. The controlled diffusion then arises as 
a formal weak limit. In obtaining the asymptotic results, this direct rela- 
tion between the queueing system control problem and the diffusion control 
problem is convenient. 



6 



R. ATAR, A. MANDELBAUM AND M. 1. REIMAN 



The policies that we estabhsh as asymptotically optimal are feedback 
controls. By this we mean that the action at each time depends only on the 
"state" of the system, namely on the number of customers waiting to be 
served and the number of customers being served, for each class. The family 
of policies among which they are proved asymptotically optimal contains 
all policies that observe all system information up to decision time. In fact, 
the family we consider is slightly broader in that the policies are allowed to 
exploit some information on the future, namely the time of the next arrival 
for each class. We comment below that this is a natural class to consider in 
the presence of renewal arrivals (cf. Section 2.2). 

Under a preemptive scheduling control, service to customers can be in- 
terrupted at any time and resumed at a later time. Consequently, the class- 
fractions of the customers waiting to be served provide natural candidates 
for control. The diffusion control problem is formulated with such a preemp- 
tive model in mind, and the control process corresponds to these fractions 
(as suggested in [21]). When restricting to scheduling control policies that 
are nonpreemptive, one must constrain the processes that count the number 
of customers routed to the server pool to be nondecreasing. The diffusion 
control problem that arises from such a model resides in a higher dimension. 
However, here we demonstrate that the nonpreemptive scheduling control 
problem is asymptotically governed by the simpler diffusion control problem 
and its HJB equation; to this end, the preemptive HJB equation is used to 
construct a nonpreemptive scheduling control policy that is asymptotically 
optimal (in fact, within the class of preemptive policies). 

Work-conserving policies are typically not optimal among nonpreemptive 
scheduling control policies. This can be seen in a simple example, where there 
are two customer classes, and the cost takes the form E /q°° e~'^* J2i=i 2 Ci^i{t) dt : ^i{t) 
is the number of class-i customers waiting to be served at time t. Consider 
the event that when the first class- 1 customer arrives, there is exactly one 
free server, and no class-2 waiting customers. If the customer is routed to 
the free server, then there is a positive probability that the class-2 customer 
that arrives next will be delayed by at least one unit of time. If the ratio 
C2/C1 is large enough, it is clear that the cost paid for delaying this indi- 
vidual class-2 customer can be larger than the cost of delaying all class-1 
customers that ever arrive (due to the discount in the cost). As a result, a 
good policy will leave a free server to idle until a class-2 customer arrives, 
or until additional servers become idle. 

On the other hand, when allowing preemptive policies, for most costs of 
interest it is intuitively clear that work conservation is optimal. We refer to 
such costs as work encouraging (see Section 5). While there is no attempt 
here at a rigorous analysis of work encouragement (this seems to require a 
different modeling framework), our results do reduce the problem of asymp- 
totic optimality (under preemption or nonpreemption) to verifying that work 
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conservation is optimal among preemptive policies (Corollary 1). For exam- 
ple, when optimality of work-conserving preemptive policies holds for the 
prelimit problems, our results, which establish asymptotic optimality of a 
nonpreemptive policy that is work conserving, imply that the phenomenon 
described in the previous paragraph is negligible on the diffusive scale. 

We comment that, to prove asymptotic optimality, it is not necessary 
to establish weak convergence of the controlled processes to a controlled 
diffusion, but only convergence of the costs. However, under appropriate 
regularity conditions of the coefficients (such as Lipschitz continuity of the 
function used to define the optimal Markov control policy; see Theorem 3), 
convergence of the controlled processes follows from our analysis. 

Diffusion control problems that arise in "conventional" heavy traffic of- 
ten have a particularly simple solution, in the form of a static priority pol- 
icy. Moreover, these policies typically exhibit pathwise minimality of the 
associated workload processes. Such a simplification is a consequence of a 
state-space collapse [19, 35] namely that these multidimensional diffusion 
control problems reduce to one-dimensional problems: in conventional heavy 
traffic, the many servers work in concert as though they constitute a single 
"super-server." While such collapse prevails in the special case studied in [2], 
simulations and intuition indicate that, in general for the QED regime, an 
analogous phenomenon is unlikely to occur. Significantly, though, our analy- 
sis does yield some state-space collapse: it is manifested through the asymp- 
totic optimality of nonpreempting work-conserving feedback controls, within 
the far broader class that allows nonpreemption, idleness in the presence of 
waiting customers and the use of all past information. 

1.4. Organization and notation. In Section 2 we describe the model, in- 
troduce a notion of scheduling control policies that do not anticipate the fu- 
ture and specify the heavy-traffic assumptions and scaling. We state our first 
main result regarding the diffusion control problem (Theorem 1). We then 
use the diffusion control problem to construct two sequences of scheduling 
control policies (preemptive and nonpreemptive) for the queueing system, 
and state our second main result on asymptotic optimality of these sequences 
of policies (Theorem 2). Section 3 treats the diffusion control problem, prov- 
ing existence and uniqueness for the underlying HJB equation, and existence 
of optimal Markov control policies. The asymptotic optimality results are 
proved in Section 4. In Section 5 we discuss the implications of our main 
result to sequences of policies that are not necessarily work conserving, and 
chart possible directions for further research. Finally, some auxiliary results 
are proved in the Appendix. 

For X £ M'^ we let ||a;|| = J2i \ Associated with the parameters k and 
n of the queueing system are the sets K = {1, . . . ,k} and N = {1, . . . ,n}. We 
write N = {1, 2, . . .}, = {0, 1,2,.. .}'', = [0, 00)^= and S'^ = {x G : 
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We denote by B{m,r) an open Euclidean ball of radius r about m. B{A) de- 
notes Borel fj-field of subsets of A. C"^''^{D) [resp. C"^{D)] denotes the class 
of functions on Z) C M'^ for which all derivatives up to order m are Holder 
continuous uniformly on compact subsets of D [continuous on D]. Cpoi(K'^) 
denotes the class of continuous functions / on M'^, satisfying a polynomial 
growth condition: there are constants c and r such that <c(l+||x||^), 
X G M*^. We let C^f = Cpoi n C™'^ For E a metric space, we denote by B{E) 
the space of all cadlag functions (i.e., right continuous and having left lim- 
its) from to E. We endow 3(E) with the usual Skorohod topology. All 
processes we consider are assumed to have sample paths in D(i?) (for appro- 
priate E, mostly = M'^). If X", n G N and X are processes with sample 
paths in I}{E), we write X" =^ X to denote weak convergence of the mea- 
sures induced by [on D(£')] to the measure induced by X. For any cadlag 
path X, let Xj„ = hm,|j X^ for t > 0, Xq- = Xq, and AXt = Xt- Xj_. If X 
is a process (or a function on M+), ||X||* = supo<s<t 11-^^(5)111 and if X takes 
real values, |X|* = supQ<j,<j |X(s)| . X{t) and Xt are used interchangeably. 
Vectors in R*^ are considered as column vectors. We write ]1 = (1,...,1)'sM'^. 
For vectors n, -y G M*^, let u-v denote their scalar product. Finally, c denotes 
a positive constant whose value is not important, and may change from line 
to line. 

2. The controlled system in the QED regime and its diffusion approxima- 
tion. We consider a queueing system which consists of k customer classes 
and n multiskilled servers (see Figure 1). Service to any customer can be 
provided by any of the servers indifferently. The service time distribution de- 
pends on the customer class, but not on the individual server (or customer). 
We say that a customer is in queue i at time t if the customer is of class i, 
and at time t it is in the system and is not being served (although it possibly 
received partial service prior to time t). Customers enter the system at one 
of the queues, and leave the system in one of two ways: either when their 
service is completed, or while they are waiting at their queue and decide to 
abandon the system without being served. 

2.1. The stochastic model. Let a complete probability space, (0, -F, P) be 
given, on which all the stochastic processes below are defined. Expectation 
with respect to P is denoted by E. The parameter n, denoting the number 
of servers, which is particularly significant in our analysis, will appear (as a 
superscript) in the notation of all basic stochastic processes associated with 
the queueing system. 

For i £ K, the number of class-i customers in the queue at time t > is 
denoted by ^2{t), and $"(t) = {^i{t), ^^i^))' ■ The number of class-i cus- 
tomers being served at time t is denoted by ^'"(t) and ^'"(t) = {"^^{t), . . . , ^'^(t))'. 
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Clearly these processes take integer values, and 

(1) gM^, ^^,"(t)<n, t>0. 

i 

The initial conditions of the system are assumed to be deterministic and are 
denoted by $"(0) = = ($?'\ . . . , $°'^)' and ^'^(0) = ^-O'" = (^-i'", . . . , 

Let A^, i £ K, be independent renewal processes defined as follows. For 
i £ K, let there be a sequence {lJi{j),j G N} of strictly positive i.i.d. ran- 
dom variables with mean EUi{l) = 1 and squared coefficient of variation 
Var([/i(l))/(£;f7,(l))2 = C^, G [0,oo). Let 

(2) Unj) = ^U,{j), ieK,jGn, 
where A" > 0. With the convention = 0, define 

(3) Ar(t)=sup|m>0:f;f/f(i)<t|, ieK,t>0. 

The value A^{t) denotes the number of arrivals of class-i customers up to 
time t. Note that the first class-i customer arrives at C/"(l), and the time 
between the (m — l)st and mth arrival of class-i customers is C/"(m), m = 
2,3,.... 

The service time of a class-i customer is assumed to be exponentially 
distributed with parameter /i", regardless of the service provider. This is 
captured in the following description. For i £ K, let be a Poisson process 
of rate /if G (0,00), and assume that the processes 5" are independent of 
each other and of the processes Af, i £ K. Let T"(t) denote the time up to 
t that a server has devoted to class-i customers, summed over all servers. 
Clearly, 

T;"(t)= f^ns)ds, iGK,t>0. 
Jo 

Then = S^{£'^2{s)ds) denotes the number of service comple- 

tions of class-i jobs, by all servers, up to time t. Our assumptions on T" will 
ensure that, for each t, T'^{t) is independent of any increment of the form 
5"(T"(t) + s) - 5"(T"(t)), s>0 (cf. Definition 2). 

For i £ K, individuals abandon queue i at rate 6*" G [O,cxo). Let be 
Poisson processes of rate 6^, independent of each other and of the pro- 
cesses A^,SJ, j G K. Note that the time up to t that a class-i customer 
spends in the queue, summed over all customers, is equal to /q <l>"(s) ds. 
Then (Jq (s) ds) denotes the number of abandonments from queue i 
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up to time t. Under an appropriate assumption on /Q$"(s)ds, similar to 
that on T" (cf. Definition 2), this describes abandonment of class-z cus- 
tomers according to independent rate-Of Poisson clocks, each run as long as 
the customer is in the queue. 

We would like to have equations that hold for both nonpreemptive and 
preemptive resume policies. Consider the processes i £ K, described 

as follows. -6^(0) = 0; Bf- increases by 1 each time a class-i job is assigned to 
a server (to start or resume service), and decreases by 1 each time such a job 
is moved back to the queue (in a preemptive-resume policy). Note that in 
a nonpreemptive policy, Bf{t) is the number of type-i customers that have 
been routed to the server pool at any time up to t. In fact, we do not assume 
that these processes only jump by ±1; their increments can take arbitrary 
values in Z. Following are the system equations: 



(4) 



These equations hold regardless of assumptions on the policy as to whether 
it is preemptive or not, and work conserving or not (these terms are, in fact, 
made precise later in this section). Note that the representations above in 
terms of Poisson processes 5" and Rf exploit the exponential assumptions 
on service times and abandonment. 

Assume that there is a full P- measure set under which all A2{t) < oo for 
t > 0, A'^ increases to infinity, AAf(t) G {0,1} for all t, and where similar 
statements hold for Sf" and R^. Then, without loss, we omit from subsequent 
discussions all realizations (sample paths) of these processes that do not 
adhere to these conditions. 

Let 

(5) X"(t) = $"(t) + ^"(t) 

and denote X^'"- = + ^f^'". Then Xf{t) is equal to the number of class-i 
customers in the system at time t. The constraints (1) can be written in 
terms of X" and as 

(6) X"(t)-^'"(t)G]R^, ^'^(i)GM^, ^^^(t)<n, i > 0, 

i 

while the system equations (4) imply that 

X^{t) = + A^it) - R^ ( (Xris) - ^ns)) ds^ - 5r ( ^Us) ds 
(7) 

ieK, t>0. 
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2.2. Scheduling control policies. We define two types of control problems, 
one where scheduling is preemptive and one where it is nonpreemptive. Equa- 
tion (7) serves as the description of the system dynamics. The scheduling 
control policy (SCP) will be identified with the process and it will be 
assumed that it is such that the constraints (6) are satisfied. Apart from a 
nonanticipating assumption on (Definition 2), there will be no further 
restrictions for preemptive scheduling control problems. For nonpreemptive 
scheduling control problems, a further constraint will be that the process B"" 
is nondecreasing in each component. 

For the following definition, note that, given a process if there exists a 
process X" so that (7) holds, then it is unique (as can be argued by induction 
on the jump times of the processes A^, and 5""). Thus (5) uniquely 
determines and either part of (4) then uniquely determines B^. Also, 
finiteness of the integrals appearing in (4) and (7) follows from the fact that 
*f are bounded by ra, while X^{t) - ^-f (t) = $f (t) < XO'" + ^J'(t). 

Definition 1. (i) We say that is a preemptive resume scheduling 
control policy (P-SCP) if it is a stochastic process with cadlag paths, taking 
values in M'^, for which there exists a process (referred to as a con- 
trolled process) satisfying the system equations (7), and such that the con- 
straints (6) are met. Given a P-SCP and a controlled process X", denote 
by and -B" the processes uniquely determined by (4) and (5). 

(ii) We say that is a nonpreemptive scheduling control policy (N-SCP) 
if it is a P-SCP, and in addition, i?", i £ K, have nondecreasing paths. 

We collectively refer to P-SCPs and N-SCPs as scheduling control policies 
(SCPs) (although the class of SCPs is simply the class of P-SCPs). 

We need a notion of SCPs that do not anticipate the future. To this end, 
denote 



stand for the time of the first arrival to queue i no earlier than t. Set 



(9) :F^ = a{Ans),SnTns)),RnTr (s)) , '^^s) , ^is) , (s) : i G K,s<t} 




and for i £ K, let 



rf(t)=inf{n>t:Ar(n)-^r("-)>0} 



and 



(10) 
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While J-p represents the information available at time t, constitutes 
future information. Since for each i, Af is a renewal process, its increments of 
the form that appears in the definition of are independent of a{Af{s) : s < 
t}. However, the time of the next arrival may be anticipated, to some 
degree, from the information on the arrivals up to time t. Therefore, with 
r"(i) replaced by t in its definition, ^" would not be a good candidate 
to represent innovative information. Note that an analogous treatment of 
and ii" is not necessary, since these are Poisson processes which are 
memoryless. The following definition refers to both types of problems. 

Definition 2. We say that a scheduling control policy is admissible if: 

(i) for each t, is independent of Qf; 

(ii) for each i and t, the process SfiT^'^it) + ■) - S^{Tp{t)) is equal in 
law to (•), and the process Rf{f^^ (*) + •)- RfiT^"" (*)) is equal in law to 

Some SCPs considered in this paper will be constructed by setting 
(11) ^'^(t) = F(X"(t)), t>0, 

for an appropriate choice of F. As the following result shows, this leads to 
admissible SCPs. 

Proposition 1. Fix n and let a function F-.Z^^Z'^ be given such 
that, forX e Z^, one hasX-F{X) G andl-F{X) < n. Then the system 
of equations (7) and (11) has a unique solution, and is an admissible 
SCP. In particular, if the process determined via (4) has nondecreasing 
paths, is an admissible N-SCP. 

See the Appendix for a proof. 

2.3. QED scaling. We consider a sequence of queueing systems as above 
where now the number of servers n € N is used as an index to the sequence. 
It is implicitly assumed that there is an SCP associated with each queueing 
system. It is assumed (without loss) that there is one probability space, 
(0, F, P), on which the processes associated with the nth system are defined, 
for all n G N. The heavy-traffic assumptions are as follows (cf. [15, 17, 21, 
34]). 

Assumption 1. (i) Parameters. There are constants Aj,//j G (0,oo), 9i G 
[0,oo), Aj,/ii G M, i£ K, such that 

k 

^Xi/l-H = 1 

i=l 
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and, as n — > oo, 

n^/\n-^X: - Xi) ^ A„ n^'^^ji: - /i,) ^ A.- 

(ii) Initial conditions. There are constants (f)i G [0, co), ■i/'j G M, f G -ftT, such 
that ipi < 0, and, with pi = Xi/ni, as n — > cxd. 

Remark 1. The above scahng is in concert with that in [12, 15, 17, 24]. 
For a verification, let denote the traffic intensity of our nth system. Then 
= r^/n, where its offered load r" is given by 

i=l 

From Assumption 1 it now follows, via simple algebra, that 

k 

Vn{l - p") ^{Pifti - Xi)/pi. 

i=l 

Denoting this last limit by (3, we deduce that 

QED scaling thus leads to square-root safety staffing [7] , which characterizes 
the regimes in [12, 15, 17, 24]. (/? > was required in the original Halfin- 
Whitt regime of [17], to guarantee stability when there is no abandonment. 
Our analysis, however, covers all values of /3 since it does not require stability 
of the queueing system. Indeed, the total discounted costs are always finite 
in view of our polynomial growth constraints on the cost functions.) 

For more details on QED scaling, readers are referred to [15] and [17]. An 
instructive comparison of the QED regime with conventional heavy traffic, 
in the context of our problem, is provided by [21]. 

The rescaled processes are defined as follows: 

xf{t) :=i.r(t)+^r(i)=^-'^r(o, 

^^{t) =nV2$-(t)=n-V2cl,-(t), 

Xnt) := ^nt) + ^nt) = n^'\X^{t) - P^) = n-'/\xr{t) - p,n). 
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The primitive processes are rescaled as 

Ant) = n-'/^A^t) - X^), Snt) = n-^'\s:{nt) - nfi^), 
Rnt) = n''/\Rtint)-nen). 
Finally, 

With this notation, the system equations (4) can be written as follows: 
= $0'- + Afit) + n^/\n-^X^ - Xi)t 



(12) 



We have from (12) 

(13) xrit) = + nwnt) + i^t - f ^r(s) ds - f ^^s) ds, 

Jo Jo 

where we denote 

r.Wnt) = A^t) - ( 1^ ^ns) ds^ - ( ^* ^ns) ds^ , 

(14) ° ° 
r. = {KCl, + X.)'/' 

and 

= nV2(n-iAr - Xi) - p^n^^fi^ - Pi). 

We now present a formal derivation of the limiting dynamics, as described 
by a system of controlled SDEs. The actual relation to the sequence of 
queueing systems (as a limit) will be justified once our results of Section 4 
are established. To this end, we pretend that the convergence 

holds, and write A, S, <I>, ^, X, B for the formal weak limits of A", 5", 

(without worrying at this point about whether weak limits exist). 
For i £ K, the processes Ai and Si are Brownian motions with zero drift 
and variances XiCfji and pi, respectively. We thus obtain 



(15) 



^i{t) = + Ai{t) + Ait - Bi{t) - Oi /* (^i{s) ds, 

Jo 

^i{t) = + Bi{t) - p\''^Si{t) - Pi f ^i{s) ds - piPit. 

Jo 



SCHEDULING IN HEAVY TRAFFIC 



15 



The corresponding constraints are as follows: 

i 

Writing W={Wi,..., Wk)', Wi = r,"^(Ai - p^^Si), the process W is a stan- 
dard A;-dimensional Brownian motion. The process X = $ + ^ then satisfies 

(16) Xi{t)=Xi + riWi{t)+U-9i f{Xi{s)-^i{s))ds- Pi f'^i{s)ds, 

Jo Jo 

as well as the constraints 

(17) X^{t)-^,{t)>0, J2^i{t)<0, 

i 

where 

ii = Xi- PiPi, Xi = (f)i + il;i. 

2.4. Work conservation and cost. A policy is work conserving if there 
can be no idling servers when there are customers in the queue. For the 
following definition, recall that 1 • equals the number of customers in all 
queues, and that 1 • equals the number of customers in the system. 

Definition 3. We say that an SCP is work-conserving if 

(18) (l-X"(t)-n)+ = l-r'(i), t>0. 

Note that equivalently 

(19) (l-X"(t))+ = l-4>"(t), t>0. 

For a given SCP, let and denote the rescaled processes as before. 
We consider the problem of infimizing an expected cumulative discounted 
cost of the form 

POO 

(20) C'' = E e-^'L{^''it),^''{t))dt, 

Jo 

over all work-conserving admissible SCPs. Under the assumption that SCPs 
are work conserving, it is more convenient to work with the function L:W^ x 
M+ defined as 

(21) L{x,u)=L{{l-x)+u, x-{l-x)+u). 

If work conservation holds, (1 • X" — n)"*" is equal to the number of customers 
waiting in all queues, namely 1 • If € S'^ denotes the proportion of 
customers of the different classes that are waiting in the queues, then 

(22) $" = (l-X" -n)+n", ^" = X" - (1 • X" - 72)+^". 

Hence (21) is merely a change of variables from ($",^") to (X",m"). The 
following will be assumed on L and L. 
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Assumption 2. (i) L(x, u) > 0, (x, u) g M*^ x S'^. 

(ii) The mapping {(t),ip) i— > L{(j),ip) is continuous. In particular, the map- 
ping (x,u) I— > L{x,u) is continuous. 

(iii) There is ^3 G (0, 1) such that, for any compact A C M'^, 

\L{x,u) — L{y,u)\ < c||x — 

holds for li G S'^ and x,y G A, where c depends only on A. 

(iv) There are constants c > and > such that L{x,u) < c(l + 
llxlp-c-), uGS'', xGM'^. 

By applying an analogous change of variables to the state equations, both 
for the queueing system and for the diffusion, one can obtain these equations 
in a new form as follows. Equation (13) for X" under work conservation takes 
the form 

(23) X4" = X°'" + riWr+ r6"(1^0ds, 

Jo 

where 

(24) 5"(1, n) = r + (//" - r)(l • X)+u - 

and r = diag(ri;i G K), T = (£y, . . . ,£^)', ^" = diag(/x^; i G i^), ^" = diag(ef ; 
i G X). Similarly, (16) for the diffusion model is now given as 

(25) X{t)=x + rW{t)+ f\{X{s),u{s))ds, 

Jo 

where for X G M^' and u G S'^, 

(26) b{X,u)=£+{^i-9){l-X)+u-fiX, 

and £ = (^1, . . . jik)' , /U = diag(/ij;i G K) and = diag(0i;i G K). 

2.5. Diffusion control problem. Below we formulate a stochastic control 
problem for the minimization of 

POO 

C = E e-"'^L{X{t),u{t))dt, 
Jo 

where X is a controlled diffusion given by (25) over an appropriate class of 
control processes u, taking values in S^. We then state our first main result 
that there exists a measurable function /iiM'^ — > S'^ such that, upon setting 
ut = h{Xt), t > 0, the infimum in the problem is achieved. 

Definition 4. (i) We cah 

7r={Q,F,{Ft),P,u,W) 

an admissible system if: 
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1. {^,F, {Ft),P) is a complete filtered probability space, 

2. n is a S'^-valued, F-measurable, (Ft) -progressively measurable process, 
and is a standard A;-dimensional (Ft)-Brownian motion. 

The process u is said to be a control associated with vr. 

(ii) We say that X is a controlled process associated with initial data 
X € M*^ and an admissible system vr = {Q, F, {Ft),P, u, W), if: 

1. X is a continuous process on {0,,F,P), F-measurable, (-Ft)-adapted, 

2. Jq \b{X (s) , u{s))\ ds < oo for every t > 0, P-a.s. [recall that b is defined 
in (26)], 

3. 

(27) X{t) = x + rW{t)+ f b{X{s),u{s))ds, < t < oo, 

Jo 

holds P-a.s. 

Proposition 2 shows that there is a unique controlled process X associated 
with any x and vr. With an abuse of notation we sometimes denote the 
dependence on x and vr by writing in place of P and in place of E. 
Denote by IT the class of all admissible systems. 

Proposition 2. Let initial data x G and an admissible system vr G 11 
be given. Then there exists a controlled process X associated with x and vr. 
Moreover, if X and X are controlled processes associated with x andn, then 
X{t) = X{t), t>0, P-a.s. 

For a proof see the Appendix. 

For X G M*^ and tt G H, let X be the associated controlled process, and 
consider the cost function 

C{x,^) = El e-^^L{X{t),u{t))dt. 
Jo 

The value function V for the control problem is defined as 

V{x) = inf C{x,n). 

Definition 5. Let x G M'^ be given. We say that a measurable function 
is a Markov control policy if there is an admissible system vr and a 
controlled process X corresponding to x and vr, such that Us = h{Xs), s>0, 
P-a.s. We say that an admissible system vr is optimal for x, if V{x) = C{x, vr). 
We say that a Markov control policy is optimal for x if the corresponding 
admissible system is. 



18 



R. ATAR, A. MANDELBAUM AND M. 1. REIMAN 



The following constitutes a part of the first main result of this paper. Its 
full version that also characterizes the value function V as the solution to 
an HJB equation, Theorem 3, is stated and proved in Section 3. 

Theorem 1. Assume L is continuous and satisfies Assumption 2(i), 
(iii) and (iv) . Then there exists a Markov control policy, /i : M'^ — > S*"' , which 
is optimal for all x S M'^. 

Throughout, h denotes the function from Theorem 1. 

2.6. SCPs emerging from the diffusion control problem. We formulate 
three SCPs that are based on the function /i, and state our second main 
result, namely that these policies are, in an appropriate sense, asymptotically 
optimal. 

A P-SCP. For each t, ^"^{t) will be determined as a function of X'^{t) 
only. Given X^[t), the diffusion control problem suggests setting 

(28) = (1 • - n)+/i(X"(t)), 
where as before 

There are two points, however, to which one must pay attention. First, 
the components of must be integer- valued, in order to represent queue 
lengths; and second, the components of = X" — must be nonnegative, 
so that one serves only those customers present in the system. 
For the first point, we need any measurable map 
that preserves sums of components and introduces an error uniformly 
bounded by a constant, so that 

(29) $"(t) = e[(l • X"(t) - n)+/i(l'^(t))] 

can be used in place of (28). For concreteness, take the following map. 
For y G M^, write yi = [y^J + 5i, and set z = Q{y) defined as Zi = , i = 
1, . . . , A; — 1, and Zk=yk + Y^^Zi ^i- Clearly, 1- z = 1-y, and z G Z^ whenever 
1 • y G Z. Moreover, ||y — z\\ < 2k: 

(30) ||e(y)-y||<2A:, yGM^. 

For the second point, note that (29) might set = X" — in such a way 
that is not in R^. For example, if X" = (n + l)ei and /i(X") = 62, then 
$2 = li which means that ^2 — Such a problem does not occur if 

(31) xr(t) > (1-X"(t) -n)+ yieK. 
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When the problem does happen, the pohcy may be defined quite arbitrarily, 
subject only to being work conserving. For concreteness, when (31) is not 
met, we set in accordance with a priority policy, where class i receives 
priority i (the higher i, the higher the priority). When (31) is met, we set 
^>"(t) as in (29). Finally, set ^'"(t) = X"(t) - ^>"(t), or equivalently, ^"(t) = 
X"(t) — $"'(f). One verifies that the constraints (6) hold by construction. 

We remark that the results of Section 4 will establish that (31) typically 
holds. This is basically due to the fact that the RHS, which represents the 
total number of customers waiting to be served, behaves at most as 0(?i^/^), 
while the LHS, representing the number of customers at each class, is 0(n). 

We next describe two alternative rules for determining sequences of N- 
SCPs. 

N-SCP (i). To describe an N-SCP for each n, one needs to determine 
so that the process B^{t) is nondecreasing. We describe a work-conserving 
SCP. A customer that arrives when there is a free server is instantaneously 
routed to a server. When a server becomes free, and there is at least one 
customer in the queue, we use the following scheme to determine which class 
to route to the server. This is in fact all that is to be determined. We look 
again at 

M"(t) := (1 • X"(t) - n)+/i(X"(i)), 

and consider the set i e K for which <^2{t) > M^^it) V 1. Note that 

if there is at least one i ^ K with > 1, then is not empty. Indeed, 
suppose that K° is empty, and let K' = {i e K : (t) > 1}. Then for i G K', 
$^(t) < Mf (t). Hence by (18), 

1 • = 1 • = ^ < ^ < 1 • M", 

a contradiction. We now choose the largest i in K^. Then a customer of class i 
is routed to the free server. This procedure is performed instantaneously. 

In heuristic terms, the scheme described above attempts to drive the 
system towards nearly achieving an equality of the form (28). This is done by 
sending to service customers of classes i for which > M", thus obtaining 
approximate equality between the quantities <I>" and M". A justification of 
this heuristic is a part of the proof of the result below. 

N-SCP (ii). The N-SCP is defined precisely as the N-SCP (i), except 
that, for each n, the function h is replaced by a function which may 
vary with n. 

By defining the interarrival times U-^{j) via lJi{j) [cf. (2)], we have as- 
sumed that they have finite variance. Here we strengthen this assumption. 
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Assumption 3. Let be as in Assumption 2. Then there is a constant 
mu > 2, mu > rriL, such that E{Ui{l))"^'^ < oo. 

Our second main result is as follows. 

Theorem 2. Let Assumptions 1-3 hold. Let X^'"- e rr^l'^Tlt- he a se- 
quence converging to x ^ . Let a sequence of work- conserving admissible 
SCPs he given, consider the corresponding processes ^>", and let ^>",^'" 
denote the corresponding resettled processes. 

(i) Let iI/"'*,<^"'* be tt sequence as determined by the proposed P-SCP 
above, and let and ^"'* be the corresponding resettled processes. Then 



Moreover, the left-hand side is finite. 

(ii) Assume thttt the restriction of h to X := {y € M'^ : 1 • y > 0} is lo- 
cttlly Holder continuous. Let be tt sequence as determined by the 
proposed N-SCP (i) and ^f'"'* be the corresponding resettled processes. 
Then (32) holds. 

(iii) Assume thttt the mttpping u L{x,u) is convex on for ettch 
X ^M.^. Then there exists tt sequence of functions {hn} with the following 
property. Let be a sequence as determined by the proposed N-SCP 
(ii), using the functions {hn}, and $"'*,^'"-'* be the corresponding resettled 
processes. Then (32) holds. 

Item (i) of Theorem 2 establishes asymptotic optimality of the proposed 
sequence of preemptive SCPs, within all work-conserving SCPs. Item (ii) 
establishes asymptotic optimality of the proposed sequence of nonpreemp- 
tive SCPs, within all work-conserving SCPs, under the assumption that the 
function h is locally Holder continuous. In Proposition 3, we show that under 
some strict convexity assumptions on L, h is locally Holder continuous, and 
thus item (ii) applies. However, for linear costs, as L{^, ^) = c • ^ (c S M*^ a 
constant), the resulting h is discontinuous (see [21]), and this part of the the- 
orem does not apply. Assuming only convexity of L(x,-), for each x (which 
certainly holds for linear costs), item (iii) establishes asymptotic optimality 
of the proposed nonpreemptive SCPs, where h is replaced by a sequence of 
functions /i„ that are locally Holder continuous. Indeed, in Section 2.7 we 
discuss additional costs of interest, where L{x,u) is convex for each x, 
implying that (i) and (iii) hold. 



(32) 
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Remark 2. The theorem is estabhshed by comparing both sides of (32) 
to the optimal cost in the corresponding diffusion control problem, denoted 
in Section 3 by V{x). It is established below that, in fact, the left-hand side 
of (32) is equal to V{x). 

Remark 3. As discussed in Section 5 (Corollary 1), for a sequence of 
N-SCPs that are not necessarily work conserving. Theorem 2 still holds given 
that work conservation is optimal among P-SCPs. 

2.7. Costs of interest. The following result provides an example for a 
family of costs for which the assumptions on h made in Theorem 2(ii) hold. 
It is proved in the Appendix. 

Proposition 3. Let Assumption 2 hold, and assume that L is of the 
form L(<I>,^) = J2ieK9i{^i)> where, for each i £ K, gi: [0,oo) — > M is in 
C^([0,oo)), and there is a constant cq > such that g'/ > cq. Then the re- 
striction of h to X is locally Holder continuous. 

Note that one can take in the above result gi{x) = CiX^'-, a > 0, Pi> 2. 

In the sequel we give examples of costs of interest, and specify the as- 
sumptions under which our main results apply. In all the cases below, L and 
L satisfy Assumption 2. Hence our results show asymptotic optimality of 
the proposed policies among work-conserving admissible policies. 

Queue lengths. Let 

where i is nondecreasing as a function of ^>j, for each i. It is assumed that 
^ > is locally Holder continuous and satisfies a polynomial growth bound. 
Then 

L{X,u) = i{{l- X)+u). 

Abandonment. We need the following result, the proof of which is given 
in the Appendix. 

Lemma 1. Under the assumptions of Theorem 2, EKi{f^ (t)) = OiE 
it). 

The number of abandonments from queue i up to time t, normalized by 
^/n, is given by 

m) ■.=n-'/'Rnfrit)). 
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Consider the cost 



i 

(the dependence of on the SCP is not indicated in this notation). Inte- 
grating by parts, using e~'^^ERf{t) ^ as f — > and Lemma 1, 

C'' = Y,-fCiE e-^'R'^{t)dt 

POO rt 

= VTQ^i^ / e-^* / ^'^{s)dsdt 
: Jo Jo 



E 



-It 



dt. 



This is a special case of the queue-length cost considered in the previous 
paragraph. 

Delay. For each of the customers / ever present in the system, let cl(/) 
denote the class to which I belongs, and let u[l) denote the set of times at 
which customer / is in the queue. We are interested in the cost 



I Jvil) 



u{l) 

where Cj > 0, i G K, are constants. Since clearly, = 'n~^/'^J2^teu(i)j 

where the sum extends over all class-i customers /, 



-7* 



dt. 



This again can be treated within the framework of queue-length costs. 

Idling servers. The number of servers that idle at time t is given by n — 
1 • ^^(t). With an appropriate normalization and discounting, this becomes 

roo 

C'' = -E e-^*l-^'"(i)di. 
Jo 

The corresponding costs are = —1 • ^ and L{X,u) = (1 • X)~ . 

Number of customers in the system. The cost associated with the weighted 
normalized number of customers in the system is 



poo 

C'^ = E e-^^YciXi{t)dt. 
Jo , 
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3. Stochastic control and the HJB equation. 

3.1. Moment estimates. We begin with a key estimate for the results of 
this section. 

Proposition 4. For any admissible system vr, any x,x ^ M.^, and corre- 
sponding controlled processes X ( associated with x and vr ) and X ( associated 
with X and vr ), the following hold: 

(i) 

\Xt-Xt\<\x-x\{l + e''^), t>0, 

P-a.s., where the constant c does not depend on 7r,x,x and t. 
(ii) For m G N, 

E^\x{t)r < c,n(i + iixir)(i + r), t > 0, 

where the constants ci,C2, . . . do not depend on vr, x and t. 

Proof, (i) Note that \X{t) - X{t)\ < |j; - ;r| + c /g \X{s) - X{s)\ ds, 
where c is the Lipschitz constant for x^ b{x,u). The result follows from 
Gronwall's lemma. 

(h) Write ^{t) = X - {1 ■ X)+u and ^'i(t) = ^{t) ■ e^. Note that ^i{t) < 
Xi{t), and 

(33) ^M/,(t) = OA^X,(t). 

i i 

Then 

Xi{t)=Xi + riWi{t)+ f\-eiXi{s)-{iJi-ei)^i{s) + £i]ds, ieK,t>0. 
Jo 

Let Ki be the set oi i £ K, where /ij > Oi, and K2 = K \ Ki. Define, for 
each i, Tj as the unique solution (cf. Theorems 5.2.5 and 5.2.9 of [25]) to 
the equation 

Ti{t) =Xi + riW,{t) + f\-fiiT,{s) + £i] ds. 
Jo 

Then Xi — Tj is differentiable, Xi{0) — Tj(0) = 0, and for i G Ki, 
j^{Xi{t) - Ti{t)) = -9iX^ - {fii - ei)^^ + /XiTi 

>-fii{Xi{t)-T,{t)). 

Similarly, the reverse inequality holds when i G K2. By comparison of ODEs 
(Theorem 1.7 in [5]), 

(34) Xi{t)>T,{t), ieKi; X,{t)<Ti{t), ieK2;t>0 a.s. 
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If z is a vector satisfying the bounds Zi > Oj for all i, and J2i^i — ^i then 
its norm can be bounded as follows: 



(35) 



< ^(2:4 - Oj) + ||a|| < A + 2||o||. 



We have in (34) inequalities analogous to >ai, when we consider Zi = CiXi, 
where q > 0, i £ Ki, and q < 0, i £ K2. Below, we obtain an inequality 
analogous to ^ ^1 by finding an upper bound on the quantity X)/^ CiXi. 
To this end, note first that by (33) and (34), 



Ki K2 \ K / K2 



(36) 



K2 Ki 



K 

Next, let c > be so small that 1 + c(l — Oi/fii) > for all i G K2. Then also 
[1 + c(l — 6i/fii)]{Xi — "^i) > 0, and as a result. 



(37) 



-i^i-Xi) + % 



<{Xi-^,) + cXi, i£K2. 



Hence, denoting x = J2ki l^i ^^^j -Ei^a '^l^i ^^i' ^(*) = T>Ki riWi{t) - Y.K2 
and £ = X^Xi ~ ^K2 have by (34), (36) and (37) 



K2 



X + W{t) + h + r E - Ms)) - ^i(s)] ds 

+ f y.<A-^^i^Oi{^,{s) - Xi{s)) + ^,{s)] ds 



< 5 + w{t) + h + 



<x + w{t) +^t + 



E -'^^{S) + E [{Ms) - ^.(S)) + CX,(S)] 
A'l K2 



ds 



E|T.(.)|+Ec|T.(.) 
K2 



ds. 



Denoting Zi = CiXi, where Ci = ^ , i £ Ki, and q = — c/ij ^, « G i^2! we have 
from (34), (35) and the above, that, for some positive constants Ci,C2, 

Ci\\X{t)\\ < \\Z{t)\\ < i + W{t) + it + C2 1* E ds + C2Y. \Ti{t)\. 
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It is easy to show that E\Ti{t)\'^ < C(l + for some constant C, and 

since Tj are Gaussian, E\Ti{t)\"'' < Cm(l + for m = 1, 2, . . . . It easily 

follows that 



E\\X 



<c™(l + ||rE|r)(l + f 



□ 



Remark 4. We record a consequence of the proof to be used in Sec- 
tion 4. Recall (23) which holds under the work-conservation condition (19). 
Arguing analogously to the proof of Proposition 4, under (19) one obtains 



(38) 



<c 



+ / ||T"(s)||(is + ||T 



where c does not depend on n or t, and where T" is the unique solution to 



(39) 



Tnt) = X^ + rrH/,'^(t)+ / {-f.:r^{s)+£^)ds. 







3.2. Cost and value. Recall that for x G M'^ and vr G 11, the cost and value 
are defined as 



C{x,tt) = E^ 



-"'^L{X{t),u{t))dt, 



V{x) = inf C{x,Tr). 

TrSn 

We assume in this section that L{x,u) satisfies Assumption 2, except that 
part (ii) should be understood as the assumption that L is continuous (the 
notation L is not needed in this section). 

To state the next result, we need to formulate a control problem on a 
bounded domain. In the sequel, T will denote a bounded open connected 
subset of M'^ with smooth (say, C°°) boundary. Let g:M.^ x dT ^ M_|_ be a 
continuous function. For x gT and vr S 11, we define 



Cr,gix,7T) = E: 



r e-^'L{Xt,ut)dt + g{T,Xr] 
Jo 



where X is the corresponding controlled process, and 

T = mi{t:Xt ^ r}. 

We also let 

Vr,g{x) = inf Cr,g(3;,7r). 

Trgn 

Proposition 5. Assume L is continuous and satisfies Assumption 2(i), 
(iii) and (iv). Then: 

(i) There is a constant c such that V{x) < c(l -|- ||x||™^), x G M'^. 

(ii) V is continuous on M'^ . 
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(iii) Let F C M*^ be a smooth domain. Let g{t, x) = e~'^^V{x) for t>0 and 
X G dV . Then V = Vx-^g in T . 

Proof, (i) This is immediate from the polynomial growth condition on 
L and Proposition 4(ii). 

(ii) Fix an arbitrary open ball of radius 1, = B{y, 1). Let x G be given, 
and for e > 0, let vr = (0, F, {Ft),P, u, W) be such that 

C{x,TT)<V{x) + e. 

Let X be the controlled process associated with x and vr. Let X be the 
controlled process on the same probability space, associated with ir and 
some X ^ u. Denote m = tul (as in Assumption 2). Let A{T) = B{y,T^"'+^). 
Let ci(T) be the Holder constant for L on A{T). By Proposition 4(ii), 

(40) E^\\x{t)\r <c{l + t"'), t>0, zei^, 

where c = c(z^). Then for any T £ [1, oo) and t G [0, T], Proposition 4(i), (40) 
and the Cauchy-Schwarz inequality imply 

E\L{Xt,ut)-L{Xt,ut)\ 

<ci{T)E[l^j,^^^^^^^T.)^\\Xt-Xtr] 

+ {either Xt or Xt<^A{T)}i^ + ll-'^tll'" + ll-'^ill'")] 

< ci(T)(l + e'^)\\x - + c[p(T) +p(r)]^/2g(l + r"^), 

where 

p{T) = snpP{Xs i A{T)), p{T) = supP(X ^ A{T)). 

s<T s<T 
The moment bounds on \\Xt\\ imply that 

p{T)+p{T)<C2{iy)T-^'^-\ 
where C2(z^) depends on u, but not on x,x £ v. Hence, writing 03(1^) = 

C{x, vr) — C{x, vr) 

POO _ 

= E e-^\L{Xt,ut)-LiXt,ut))dt 
Jo 

< {ci(r)(l + e''^)\\x - x\\<^ + cc3{u)T-"'-\l + T'^)}E ^ e-^' dt 

Jo 

POO _ 

+ c / e-^\l + E\\Xt\r + ^ll^tir) dt 
Jt 

<Ci{T)\\x-x\\^ + C5{iy)a{T), 
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where C4(T) depends only on T, 05(7^) depends only on u, and a(T) ^ as 
T — > 00. Let T be so large that C5(i^)a(r) < e. Next choose 5 > so small 
that {0 < ||x — x\\ < 6 and x,x £ u} implies C4(T)||x — x\\^ < e. Then for 
such x,x one has V{x) < C{x,tt) < C{x,7r) + 2e < V{x) + 3e. Note that the 
choice of 5 does not depend on x,x (in particular, it does not depend on vr!). 
Therefore, the inequality V{x) < V{x) + 3e holds for all x,x £ i' for which 
1 1 a; — a; 1 1 < 6. This shows that V is continuous. 

(iii) This is a standard result (the principle of optimality), which, in the 
current context, can be proved similarly to the results of [6], Section III.l. 
□ 

3.3. The HJB equation and optimality. The HJB equation associated 
with the stochastic control problem is (cf. [11]) 

(41) £f + H{x,Df)-jf = 0, 
where C = {^/'2)J2i''"i9^/dxf, and 

H{x,p)= inf [b{x,u) • p + L{x,u)]. 

The equation is considered on M.^ with the growth condition 

(42) 3C,m, |/(x)| < C(l + ||x||"'), 2; G 

We say that / is a solution to (41) if it is of class C^, and the equation is 
satisfied everywhere in M'^. 

Theorem 3. Assume L is continuous and satisfies Assumption 2(i), 
(iii) and (iv). Then there exists a classical solution f G Cp^^(M'^) to (41), 

(42) , and this solution is unique in CpQj(]R'^). Moreover, the value V is equal 
to f. Furthermore, there exists a Markov control policy which is optimal for 
allxGR''. 

Proof. We first consider equation (41) on a smooth open bounded con- 
nected domain F, satisfying an exterior sphere condition, with boundary 
conditions 

(43) f{x) = V{x), xedT. 

The key is a result from [16] regarding existence of classical solutions in 
bounded domains, with merely continuous boundary conditions. To use this 
result, we verify the following two conditions: 

(i) \H{x,p)\ < c(l + IIpII) for 2; G r, where c does not depend on x or p. 

(ii) H{x,p) £C%Tx R''), some e G (0, 1). 
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Item (i) is immediate from the local boundedness of b{x,u) and L{x,u). 
Next we show that item (ii) holds. For 6 > 0, let v be such that H{y, q) > 
b{y, v) • q + L{y, v) — 6. Write 

H{x,p) - H{y, q) < b{x, v)-p + L{x, v) - b{y, v) ■ q - L{y, v) + 6. 

Using the Holder property of L in a: uniformly for {x,v) S F x S*^, and the 
Lipschitz property of b in x, uniformly in {x,v), 

H{x,p) - H{y,q) < c\\p - q\\ + c||p||||x - y|| + c||a; - y||^ + 6. 

Since 5 > is arbitrary, it can be dropped. This shows that H is Holder 
continuous with exponent g, uniformly over compact subsets of F x M*^. 
Hence (ii) holds. 

Defining for {x,z,p) G FxMxM'^, A{x,z,p) = (l/2)r^p, B{x,z,p) =H{x,p)- 
72;, one can write (41) in divergence form as 

d\YA{x,f,Df) + B{x,f,Df) = Q. 

The hypotheses of Theorem 15.19 of [16] regarding the coefficients A and 
B hold in view of (i) and (ii). Indeed, B is Holder continuous of expo- 
nent Q, uniformly on compact subsets of F x M x M'"'. Moreover, with r = 0, 
v{z) = (1/2) miuj r?, ^(z) = c(l + \\z\\), a = 2, 61 = and ai = 0, one checks 
that the conditions (15.59), (15.64), (15.66) and (10.23) of [16] are satis- 
fied. Theorem 15.19 of [16] therefore applies. [We comment that there is a 
typo in the statement of the conditions of the theorem in [16]: the reference 
should be to condition (15.59) instead of (15.60).] It states that there ex- 
ists a solution to (41) in C^'^(F) n C(F), satisfying the continuous boundary 
condition (43). We denote this solution by /. 

Let X G F. Let vr be any admissible system and let X be the controlled 
process associated with x and vr. Let r denote the first time X hits dV. 
Using Ito's formula for the C^'^(M+ x F) function e~'^*/(x), in conjunction 
with the inequality 

Cf{y) + b{y, u) ■ Df{y) + L{y, u) - ^f{y) > 0, y G F, n G S^ 

satisfied by /, one obtains 

rtAr 

f{x)< / e-^'L{Xs,Us)ds 
Jo 

i-tAr 

+ e-^(*^^)/(XMr) - / e-^Wf{X,) -rdWs. 
Jo 

Taking expectation and then sending t — > 00, using the monotone conver- 
gence theorem as well as the bounded convergence theorem, we have with 
git,x) = e-'''Vix), 



(44) 



fix) < El 



r e-^'L{Xs,Us) ds + e-'<^V{Xr] 
Jo 



Cr,g{x,TT). 
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Taking the infimum over vr G IT, we have 

f{x)<Vr,g{x)=V{x), xer, 

where the last equahty follows from Proposition 5(iii). 

In order to obtain the equality f = V on F, we next show there exist 
optimal Markov control policies for the control problem on T. Let 

(45) ip{x,u) = b{x,u) ■ Df{x) + L{x,u), x£T,ueS''. 

Note that ip is continuous on T x S^. For each x, consider the set Ux ^ of 
u G S'^ for which 

ip{x,u)= inf ip{x,v). 

We show that there exists a measurable selection of Ux, namely there is a 
measurable function h from (T,B{T)) to (S'',B{S'')) with h{x) £Ux, x£ T. 

Let G r and assume lim„ Xn = x gT. Let Un be any sequence such that 
Un S Ux„ ■ We claim that any accumulation point of Un is in Ux , for if this 
is not true, then by continuity of (p, there is a converging subsequence Um, 
converging to n, and there is a n such that 5 := ip{x,u) — ip{x,u) > 0. Hence 
for all m large, ip{xm-,Um) > ^{x-iu) + 5/2 > ip{xm,u) + S/A, contradicting 
Um G Ux^. 

As a consequence, the assumptions of Corollary 10.3 in the Appendix 
of [10] are satisfied, and it follows that there exists a measurable selection 
h-.T^S'' of {Ux,x£r). 

We extend h to M*^ in a measurable way so that it takes values in §^ (but 
is otherwise arbitrary). Clearly, x b{x , h{x)) is measurable. Consider the 
autonomous SDE 

(46) X{t)=x + rW{t)+ f\{Xs)ds, 

Jo 

where b{y) agrees with b{y,h{y)) on F, and is set to zero off F. Then b is 
measurable and bounded on W^. By Proposition 5.3.6 of [25], there exists 
a weak solution to this equation. That is, there exists a complete filtered 
probability space on which X is adapted and is a fc-dimensional Brownian 
motion, such that (46) holds for t > 0, a.s. On this probability space, consider 
the process Us = h{Xs). Since X has continuous paths and is adapted, it is 
progressively measurable (see Proposition 1.13 of [25]) and by measurability 
of /i, so is n. Denote by vr the admissible system thus constructed. Then for 
s <T, Us ^ Uxs and 

b{Xs,Us) ■ Df{Xs) + L{Xs,Us)=H{Xs,Df{Xs)). 

Hence 



Cf{X) + b{Xs,Us) • Df{Xs) + L{Xs,Us) - -ff{Xs) = 0, s<T. 
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A use of Ito's formula and the convergence theorems just as before now 
shows that 



r e-""L{Xs,Us) ds + e-"'W{Xr) 
Jo 



Cr,g(x,7r), xGT, 



with g as above. This, together with the principle of optimality, shows that 
/ > Vr^g = V onT. Summarizing, f = V on T. 

In particular, V G C^'^(r) and is a classical solution to the HJB equation, 
r can now be taken arbitrarily large, and this shows that V S C^'^(M'^), 
and that it satisfies the HJB equation on M'^. In view of Proposition 5(i), 
it also satisfies the polynomial growth condition. As a result, there exists a 
classical solution to (41) in C^'^(]R'^), again denoted by /, satisfying (42), 
and moreover, V = f. 

It remains to show uniqueness within C^^^(M.^) and existence of optimal 

Markov control policies for the problem on M'^. Let / G Cp^i{M.^) be any 
solution to (41). Then analogously to (44), we obtain 



/»</ e-^'L{Xs,Us)ds + e-^'f{Xt)- f e-^Wf{Xs)-rdWs. 
Jo Jo 

Taking expectation, sending t — > cxd, using the polynomial growth of / and 
the moment bounds on ||^t||, one has that f{x) < C{x,tt), where vr G 11 is 
arbitrary. Consequently, / < y on M"^. 

The proof of existence of optimal Markov policies as well as the inequal- 
ity V < f on M.^ is completely analogous to that on F, where one replaces F 
by M.^ and uses again the polynomial growth condition of /. The weak ex- 
istence of solutions to (46) follows on noting that b satisfies a linear growth 
condition of the form ||6(y)|| < x{l -\- \\y\\), y G M'^, and using again Propo- 
sition 5.3.6 of [25]. Hence V = f on M*^. We conclude that / is the unique 
solution in CpQj(M'^), that V = f, and that there exists a Markov control 

policy, optimal for all x G M^. □ 

4. Asymptotic optimality. In this section we prove asymptotic optimal- 
ity of the proposed SCPs. As in the statement of Theorem 2, all SCPs are 
assumed to be work conserving in this section. Recall from Section 2 that 
the processes and represent the number of customers waiting in each 
queue, and, respectively, the number of servers working on jobs of each class. 
Let u"' be an S'^-valued process, determined as 

(A7) ^,n_(<^^/{l-X--n)+, l-X"-n>0, 

^ ' Uo, l-A:"-n<0, 

where uq is some fixed, arbitrary element of S^. As in the paragraph pre- 
ceding Assumption 2, represents the fraction of customers of each class 
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that are waiting in the queues (whenever there are such customers). As a 
result one can determine ^>"' and from n" and X" as ^" = (1 • X^)~^u'^ 
and if" = X"-l>". 

Throughout this section let / denote the unique Cp^j solution to (41) 
(cf. Theorem 3). Let 

(48) = 6(Xr, <) • DfiXj^) + L(Xr, uD - //(Xf, Z)/(Xn) > 0. 
A condition that plays a central role in the convergence proof is 

(49) / e-^'K^ds^O. 



Theorem 4. (i) Let Assumptions 1-3 hold. Let X^'"- G n^^/^Z^' be a 
sequence converging to x £ M'^. Let a sequence of work-conserving admissible 
SCPs be given [namely, (19) holds], let X"^ be the corresponding normalized 
controlled processes starting from X^'"^ and let be given by (47). Then 

lim^inf E / e-^^L{X^, u^) dt > V{x). 

•J 

(ii) Assume, in addition, that (49) is satisfied. Then 



roo 

limsup^; / e-^*L(Xj",Odt <F(a 

n— >oo JO 



In what follows we prove Theorem 4. We treat both parts (i) and (ii) si- 
multaneously. Whenever there is a reference to part (ii), we indicate explic- 
itly that (49) holds. It will be convenient to work with both representations 
(13) and (23) for X^ in this section. Denote 

(50) Yr= f\''{x^,u':)ds, zr= f\-^'L{x^,u':)ds. 

Jo Jo 

Let (J^") be the filtration (9). Note that, by definition, the processes X", l>", 
are adapted to Hence by (47) and (50), so are the processes u", y" and Z". 

Lemma 2. Under Assumption 3, 

E/dli"!!*)'"^ <c(l+t'^^/2)^ nGN, t£R+, 

where c does not depend on n or t. 

Proof. This is a consequence of Theorem 4 of [26], which, under the 
assumption E{Ui{l))'^'^ < oo, mjj > 2, states that 

(51) Esup\n-^/\A'^{ns)-nXis)\"''' Kcil + t""^/^), 

s<t 
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where 

Ai{t)= sup i^m>0:fyJi{j)<t^, t>0, 

and c does not depend on n or t. Indeed, by (2) and (3), A'^{t) =Ai{X"t). 
Let C = sup„[A"/(nAi)] and note that C < cxd by Assumption 1. Then 

iirir=supn-vv.(A^)-Ar5i 

s<t 

< sup n~^^'^\Ai{nXis) — nXis\. 

s<Ct 

The lemma follows from (51). □ 

Lemma 3. Under the assumptions of Theorem 4(i), the processes 
satisfy E\\X'^{t)\\'^^ < c(l + + t"^), where rh and c do not depend 

on n, X or t. 

Proof. Since we are assuming work conservation, (38) applies. Solving 
for T" of (39), we obtain 

rnt) = X°'"e-^* + Wnt) - t Wns)e-^"'^'-^^ ds, 

Jo 

where 

wr{t) = r,wr{t) + e'^t. 

Hence 



(52) \\xn<c 



i + + + iiiy/^ll + / \\w'^\\ds+ / \\wj}\\deds 

Jo Jo Jo 

By (14), using ^f{s) < 1 and ^f{s) < ^"(s) := max^rj-^x"'" + n-^Af (s)], 
(53) ||I^"(t)||<||i"(t)||+sup||5"(s)||+ sup ||i?"(s)||. 

s<t s<?"(t) 

Denote p = mu. Apply Burkholder's inequality (cf. [33], page 175) to the 
(discontinuous) martingale , denoting by [M] the quadratic variation pro- 
cesses associated with M, and recalling that if a process M taking real values 
has sample paths of bounded variation, then [-/Vf](t) = Mq +^q^^<^(AM5)^. 
Denoting by (t) ^ Poisson random variable with parameter n/ift and using 
the convergence n'^ ^ fii, we obtain 

Esnp\Sf{sW<cE{[Smf' 



s<t 
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where c does not depend on n or t. Similarly, ii/sup^^j 
Therefore, by the independence of A'^ and i?" and Assumption 3, 

E sup \Rnsw=E{E\ sup iRnsweit)]] 



<cEiCit)f^^ 
<c{l + t''), 

where q does not depend on n or t . Lemma 2 and an application of Minkowski's 
inequality to (53) show that there is m not depending on n or t such that 

(54) S||#"(t)||P<c(l + t'"), t>0. 
The lemma now follows from (52). □ 

Lemma 4. Let the assumptions of Theorem 4(i) hold. 

(i) (A", 5", i?") (yl, 5, R), where A, S and R are independent Brown- 
ian motions with zero drift and variance matrices diag(AjC^ Jjgx, diag{fj.i)i£K , 
and, respectively, diag{9i)i^K ■ 

(ii) One has 

(55) (^'",^")^(p,0) in(p(R'')f 

[the process that is constantly (p, 0)]. 

(iii) The sequence (X", y", Z", TV") is tight [in (B(R'=))'^]. 

Proof, (i) By the assumption on the finite second moment and i.i.d. 
structure of the interarrival times, and by Assumption 1, the results of [22] 
imply (i). 

(ii) Since work conservation (19) is assumed, we can use (38). Note that 
X'^-p = By part (i), n'^/^^" ^ 0. Also, n-Vs^o.n ^ q_ ^^^^^ 
by Gronwall's lemma, n~-'^/^supg<;J|T"(s)|| ^ in distribution for any t, 

and therefore ?i~"^/^T"' ^ 0. As a result, n~^/^A"" 0, which implies that 
X"- ^_p. Using 1 • p = 1 and 1 • ^'^ = (1 • X" - !_)+, we have that 1 ■ l>"_=^ 0. 
Now <l>f =^ follows since <I>f > 0. Using X" = <!)" + we have that ^t" ^ 

P- 

(iii) By (i). A"' ^ A. By (i) and (ii) and a time change lemma (cf. [4]), it 
directly follows that Sf'lJ^ ^"(s) ds) ^ Si{pi •). A use of (i), (ii) and a time 
change lemma also shows that R^{Jq ^"(s) ds) =^ 0. Hence by (14), 



7-n ^ ^-1, 

where is a standard A;-dimensional Brownian motion. 



(56) Wl" r-^rW = W, 
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Since are relatively compact, they are tight. Hence by [4], Theo- 
rem 16.8, for each t, limm^oc^^^sup^_^^ P{\\W^\\t >m) = 0. By (23) and 
the Lipschitz property of the functions x 6"(x,n), uniformly in x, u and n, 

< + WW'^m +c + ||l"(s)||) ds. 

Jo 

By Gronwall's inequality, using the boundedness of X^'^, n G N, we have 

(57) ||l"|k<ce^*(l + ||W"||t). 
This shows that, for each t, 

(58) lim limsupPdlX'^llt >m) = 0. 

m— »oo ^_>oo 

Fix T. It follows from (23) that, for any s,t£ [0,T] with s <t, 

||l"(t) -X"(s)|| < WW^'it) - Ty"(s)|| +c f\l + \\X''{s)\\)ds 

J s 

(59) 

< WW^it) - W^is)\\ + c{t - s){l + W'Wt). 



Recall the modulus of continuity defined for x G D(M ) restricted to [0,T] 
(cf. [4], page 171) as 

w'rp{x, 6) = inf max w{x, tj)), 

l<i<-u 

where the infimum is taken over all decompositions [ti-i,ti), 1 <i <v, of 
[0, T) such that U - U-i > 5 for l<i<v. Here, for S C [0, T), 

w{x, S) = sup \\x{s) — x{t)\\. 

s,t&S 

By tightness of W^, Theorem 16.8 of [4] implies that, for each t and e, 
\imlimsupP{w[{W"-,5)>e) = 0. 

<5— ►O n— ►oo 

Using (59), a similar statement follows for X", namely that, for each t <T 
and e, 

(60) limlimsupP(Wf(X'',5) > e) = 0. 

S^O ji— >oo 

By (59) and (60), and since T is arbitrary, the tightness of X" follows from 
Theorem 16.8 of [4]. 

Noting that ||y"(t)|| < ct(l + ||X'^||t), and ||Z"(i)|| < ct(l + ||X"||J^) (m of 
the L), and that for s,t<T, ||y"(t) - F"(s)|| < c\t - s\{l + ||X"||t) and 
1 1 (t) - (s) 1 1 < c| t - s I ( 1 + 1 1 X" 1 1 ) , the tightness of and of follows 
from (58) using again Theorem 16.8 of [4]. □ 
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We use the following (very special case of a) result of Kurtz and Protter 
[28]. Let (Ft) be a filtration. A cadlag, (Fj)-adapted process V is a semi- 
martingale li V = M + N, where M is an (i<t)-local martingale, and the 
paths of N are of bounded variation over finite time intervals. An M'^-valued 
process is an (i<t)-semimartingale if each component is a semimartingale. 
Write / U dV for /q U{s—) ■ dV{s). A cadlag process V has bounded jumps 
if there is a constant c such that ||T^(s) — ^(s— )|| < c, s S (0, oo), a.s. Denote 
by [M] the quadratic variation process associated with M, and by Tt{N) the 
total variation of over [0, t] . 

Lemma 5. For each n, let (f/",!/") he an {F^)-adapted process with 
sample paths inD((M'^)^) and let he an {Fl^) -semimartingale with hounded 
jumps. Let = M^ + N"^ he a decomposition ofV^ into an {Fp)-local mar- 
tingale and a process with finite variation. Suppose 

(61) for each t > 0, sup£;[[M"]t + TtiN'')] < oo. 

n 

If ([/",!/") => {U,V) in the Skorohod topology on B{{M.'')^), then V is a 
semimartingale with respect to a filtration to which U and V are adapted, and 
(C/",y",/C/"(iy") {U,VjUdV) in the Skorohod topology on B{{R''f). 

Proof. The proof follows from Theorem 2.2 of [28] on taking, for a > 0, 
= a + 1, noting that V,^ = Vn if 5 is a fixed large constant. □ 

Lemma 6. Let the assumptions of Theorem 4(i) hold. Denote by {X, Y, Z, W) 
a limit point of (X", 1"", Z", TV") along a suhsequence. Let (Ft) denote the 
filtration generated hy {X,Y,W). Then W is an (Ft) -standard Brownian mo- 
tion, X , Y and Z have continuous sample paths, and Y has sample paths 
of bounded variation over finite time intervals. Moreover, J e~'^^Df{X'^) ■ 
dY^ ^ / e~^^Df{Xs) ■ dYs along the subsequence, where f is the solution 
to (41). 

Proof. The processes Y and Z have continuous sample paths since 
y" and do (see Theorem 3.10.2(a) of [10]). Since = + rW"- + F", 
and converges in distribution to a Brownian motion [cf. (56)], X = x -\- rW + Y 
has continuous sample paths. To see that Y has sample paths of bounded 
variation, write = - F"-", where YP'^{t) = /q (y"(s))+ ds, y"'~(t) = 
loi^i'is))' ds. By definition (50) of Y^' and (24) of b'', 

(g2) ^"'^W V y^'-it) < cj\l + \\Xn)ds, 

(y"'+(t) - y"'+(s)) V (y"'-(t) - y"'~(s)) < c\t - s|(i + \\x^t), 
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where c does not depend on t,n. Thus it follows from the tightness of 
that y"->~) is tight. Let {Y~^,Y~) denote any subsequential limit point 

in (D(R'^))'^. Since y"'+ and y"-'~ have continuous sample paths, so do 
and Y~ , and therefore Y = Y~^ — Y~ . Since Y^ and Y~ have nondecreasing 
sample paths, Y has sample paths of bounded variation over [0,t] for any t. 

Next we apply Lemma 5 with [/" = e"^*L>/(X"(i)), = y^, and (F/^) = 
(J^f ) of (9). By Definition 2 and the definition of ^ , clearly X" and 
are adapted to (F/^). We decompose = M" + iV" as W = 0, A^" = F". 
By (62), and Lemma 3, (61) holds. By the continuous mapping theorem, 
(e-^*D/(X"(i)),y"(t)) converges to {e""'^ D f {X (t)) ,Y {t)) in the Skorohod 
topology on (©(M^^))^. By continuity of the sample paths of Y"', it follows 
that the convergence in fact holds in the Skorohod topology on D((]R'^')^) 
(see Proposition 6.3.2 of [10]). As a result of Lemma 5, / e""'^ D f {X"^ {€)) ■ 
dY^'it) J e-^^Df{X{t)) ■ dY{t). 

It was shown in the proof of Lemma 4 [cf. (56)] that converges to 
a standard Brownian motion. To see that W is in fact an (i<t)-Brownian 
motion, note that by definition it is adapted to {Ft). It remains to show 
that, for each t, Ft is independent of a{Wt+u — : u > 0}. Fix t > 0, u > 
and < s < t. Write a"" = {X^,Y^, W^) and a = (X„ y„ W^). By (14), using 
the notation (8), and denoting 

5r = S^{n-^Tnt)) - S^:{n-^T^{t + u)), 
i?r = R'lin-^ it)) - RUn-' TP it + u)), 

we have 

r.iwnt + u)- wnt)) = Ant + u)- A2{t) -s^-r: 

where 

r,A" = AUrnt) + u)- Anrnt)) - - R^ 

and 

6^ = ir(t + n) - A^it) - A^T^it) +u) + ir(rr(t)). 

Let / : {MJ^f ^ M and 5 : M'' ^ M be bounded continuous. By (9) and (10), 
Z?" is measurable on and a" is measurable on J^". By the admissibility 
assumption and Definition 2, 

(63) Ef{a^)g{n = Ef{a^)Eg{n- 

Since T"(t) converges in distribution to zero, and A" converges in distri- 
bution to a continuous process, it follows by a random change of time 
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lemma ([4], page 151) that 6f converges in distribution to zero. As a re- 
sult, /3" converges in distribution to Wt+u — Wt- Using (63), the convergence 
(X",!^",!^") ^ {X,Y,W) and the continuous mapping theorem, it follows 
that 

(64) Ef{a)g{Wt+u - Wt) = Ef{a)Eg{Wt+u - Wt). 

By approximating indicator functions of closed sets of (M*^)^ (and resp. M'^) 
by continuous functions / (resp. g), it follows that (64) holds when / and g 
are replaced by such indicator functions. Since n > and s <t are arbitrary, 
an application of the Dynkin class theorem (Theorem 1.4.2 of [9]) shows 
that Ft is independent of a{Wt+u — Wt:u> 0}. Since also t is arbitrary, it 
follows that W is an (Fj)-Brownian motion. □ 

Proof of Theorem 4. We first prove part (ii). Recall that (49) holds. 
Let {X,Y,Z,W) be a weak limit point of (1", F", Z", #") and let (Ft) be 
the filtration generated by {X, Y, W). By Lemma 6, Xt = x + rWt + Yt, W is 
a standard (Fj)-Brownian motion and the sample paths of Y have bounded 
variation over finite time intervals. Just as before, an application of Ito's 
formula and the fact that / satisfies the HJB equation (41) give 



f{Xt)=f{x)+ f\-^'Df{Xs)-rdW, 
Jo 

+ f\~^'Df{Xs)-dYs- f\~^'H{Xs,Df{Xs))ds. 
Jo Jo 



(65) 

By (48) 
(66) 
where 

en{t) = f e-^%b{X^,u-,)-b-{X-,u:))-Df{X-)ds. 
Jo 

By definition of the functions b and 6" and by Assumption 1, 
||6(X^<) - 6"(X^<)|| < e„(l + 



e~'''K^ds = f\-^'Df{X2) ■ dY^ 
Jo 

+ en{t) + - f e-^^H{X^,Df{X2)) ds, 
Jo 



where ^ 0. Therefore, Lemma 3 and the continuous mapping theorem 
imply that e„ =^ 0. We get from (49) and Lemma 6, using continuity of 
x^H{x,Dfix)), 

(67) f\-^'Df{Xs)-dYs + Zt- f\-^'H{Xs,Df{Xs))ds = 0. 
Jo Jo 
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Combining (65) and (67), 

< e~^'f{Xt) = fix) + f e~^'Df{Xs) -rdWs- Zt. 

Jo 

Hence 

(68) Vt, EZt < fix). 

Fix an arbitrary 5 > 0. By Lemma 3 and Assumption 2, there is T such that 

(69) E J e-^'LiX^, <) ds < 6 

for all n. Since ^ Z and Z has continuous sample paths, Zj. converges 
in distribution to Zt- By Jensen's inequality. Assumption 2 and Lemma 3, 

S(Z^)^+^/™^ < cE /^e"^(^+^/™^)"(l + \\X^\r^+')ds 
Jo 

(70) 

< c, 

where c does not depend on n. Hence ZJp, n G N, are uniformly integrable, 
and one has EZj, — > EZt as n — > oo. By (68) and (69), we therefore have 
that 

/■oo 

limsup£; / e-^'LiX2,u'^)ds<fix)+6. 

n— »oo JO 

Since (5 > is arbitrary, it can be dropped, and part (ii) of the theorem 
follows. 

Next we prove part (i) . Arguing as in part (ii) but using > instead 
of (49), we have that (65) holds and 

/* e-^'DfiXs) ■ dYs + Zt- f e~^'HiXs,DfiXs)) ds > 0. 
Jo Jo 

Hence 

(71) e-^'fiXt) > fix) + [\-^'DfiXs)-rdWs - Zt. 

Jo 

By Proposition 5 and Lemma 3, 

i?/(lr)<c(i + r^), 

for t >0 and n G N. Since for each t, fiX^) converges in distribution to 
fiXf), and fiX"^) are uniformly integrable [arguing as in (70), using the 
growth condition of Proposition 5(i)], one has EfiXt) < c(l +t'"^), where 
again c does not depend on t. We therefore have, from (71), 

EZt>fix)-ait), 
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where a{t) — > as t — > oo. Note that as in part (ii), given 5 > 0, (69) holds for 
ah T large enough, and that, for each T, Zj,, n G N, are uniformly integrable. 
Hence 

liminf^ / L(X^,u':)ds>EZT-6>f(x)-a(T)-6. 

Part (i) of the theorem now follows on taking T ^ oo and 6^0. □ 

Proof of Theorem 2. We only need to show that the proposed SCPs 
satisfy the conditions of Theorem 4(ii). The work-conservation condition (19) 
holds for both of the proposed SCPs, by definition. To conclude parts (i) and (ii), 
it remains to show that in both cases (49) holds. Part (iii) is treated there- 
after. 

(i) The P-SCP. Fix T. Let 17" denote the event that (31) is met for ah 
t G [0,r]. Recah that on 17", the P-SCP sets 

$(i) = e[(i-x"(t)-n)+/i(x"(t))], tG [o,r]. 

Let 

(72) ;7" = (i-x")+/i(x"), y" = d"-[/". 

Recall that h satisfies 

b{x, h{x)) ■ Df{x) + L(x, h{x)) = H{x, Df{x)), x G R*=. 

Note that for x with 1 • a; < 0, b{x,u) is independent of u [see (26)] and so 
is L{x, u) = L{{1 • x)~^u, X — {1 • x)'^u) [see (21)]. Hence 

(73) ini [b{x,u) ■p + L{x,u)]=b{x,v) ■p + L{x,v), wGS^, 1 • x < 0. 

For t such that 1 • < 0, (73) and (48) imply that i^f = 0. Next consider 
t such that 1 • X" > 0. We have 

u"^ = $"(t)(l . X^)-^ = h{X^) + ■ X^)~^. 

By assumption, L is uniformly continuous on compacts. For each k, let 
a^((5) be such that a^{5) i as S 10, and |Z(0,?/') - L{(j)',i;')\ < a«((5) 
whenever ||(/)||, \\4>'\\, ||^||, [[V''!! < k, and \\4> — 4>'\\ V \\ip — ip'\\ < S. Then us- 
ing (21), writing = 1 • X^, the following holds on the event 0"'** : = 

m = mx^,^t) - b{x^Mxm ■ Dfixn 

+ L{{Q)+u^,X^-{i^)+u^) 

(74) , . . 

- m^)+h{x^), - {Qrh{xm 

<c\\Vn\\Df{X^)\\+a'^{c\m\)- 
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By (30), < 2A;n-i/^ As a result, < on O"'" for some e„ 0. 

The events ri" have probabihty tending to 1 as n — > oo, as follows from the 
convergence =^ p shown in Lemma 4. The tightness of (see Lemma 
4), (19) and the fact that G imply that 
(75) lim liminfP(0"''') = L 



K— >oo n— >oo 



Therefore converges to zero in distribution. Since T is arbitrary, K"^ ^ 

0, and (49) holds. 

(ii) The N-SCP (i). Fix T. Let U"" and be defined as in (72). A 
review of the previous paragraph shows that, replacing throughout by 
rj, (74) and (75) still hold. Fix eq > 0. We next estimate, for any e > 0, 

limsuppf sup ||y"(t)|| >8tey 

Fix i^K.li either V^'-{t) < or (t) = holds for ah t G [s, r), then within 
this time interval, the SCP does not route any class-i customer to the service 
pool. Therefore by (4), for t G [s,r), 

(76) l-rW = + n-^'^A^{s,t) - n-'/'Ans,t), 

where we write 

Ans,t) = Ant)-Ans), 

Given e > 0, 

\n.k 



(77) P[ inf y,"(t) < -4e <P((J7"''^)^) + P(17^''=) + P(17^'"), 
where 

^ ^n,K n ja £0 < s < r < T : V^is) > -e, 

sup vr{t)<-e,Vr{r)<-Ae], 

= 0"''= n ( sup Vr{t)<-e\. 

Using the local Holder property of h on X, for any k, there are > and 
Pk G (0, 1] such that, on O"''', 

\U:{t)-Ul^{s)\<c,\\X-{t)-X^{s)r 

+ (^/'^)l{l.X"(s)<e/8} + (^/^)-'-{l-X"(t)<e/8} 
<C,||X"(t)-X"(s)|r +e/2. 
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Writing 

n-i/2A^(s,t) 

and using ||^"||^ < nn~^/'^ on ri'"'**, we have 

n-i/2|Ar(s,t)|<2||i?"||;:^„_,/, + cK(t-.). 

Hence on il"''^, for t G [s,r), 

- cjx^it) - x-{s)r - 2iii?"ii:T.-i/2 - c^it - s). 

On Qi'^ we also have Vl^{r) < — 4e. Let /3 > and write f3 = /?+ck. Therefore 

(78) P(J7^''^) < ) + ^(^^"'2 ) + Pi^l^), 

where 

O^f = {3 0<s<r<r: n^^/^^" (s, r) < -e + /3(r - s)}, 

^TS = {3 < s < r < T : c,||l'^(r) - X^isW^ > e + /3(r - s)}, 

^^?:3={2||i?^|i:^„-V2>e/2}. 

Using the monotonicity of the processes A" and the uniform convergence of 
n~^A^ on [0,T] to Ai{t) = Ajt, as follows from the convergence A" ^ A [see 
Lemma 4(i)], 

limsupP(r2"'f ) 

n ' 

<limsupP(30<s<r <r:r-s>e//3, n~^A"(s,r) < n~^/2/3r) 

n 

(79) 

< limsupPf sup ||n"^A"(i) - Aii|| > c 

n \t<T 

= 0. 

Also, 

P(0";2'') <P(30<s<r<T,r-s> /J'^/^ . c^||X"(r-) - X'^(s) f'' > /?i/2) 
+ P(3 < s < r < T, r - s < /J'^/^ . c^||x"(r) - X"(s)f > e) 
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By Lemma 6, the processes X" are tight and converge along subsequences 
to processes with continuous sample paths. Therefore 

(80) lim limsupP(0"'2^) = 0. 

The convergence of i?" to a Brownian motion (Lemma 4) implies 

(81) limP(O^;3")=0. 

By a similar argument, on i^^''^, 
VnO) + n-y^A^eo) - c.||l"(eo) - X"(0)r'= - e/2 - 2||i?"||*^„_,/, - cksq 
<Vr{eo)<-e. 
Hence, for some constant c^, 

^^^^ limP(Or) <limP(n-Mr(eo) <c>-i/2 + 2n-V2||^n||*^^_^^^) 
= 0, 

where the last equality follows from the convergence in distribution of n~^Af{eo) 
to AjSo a-nd of R"' to a Brownian motion. Combining (77)-(82) shows that 



limsupP inf ^"(t) < -4e < limsupP((0"'^)^). 

n \te[eo,T] J n 

Note that by (19), 1 • = 0. Hence = 2(1 • F")". Since i € is 
arbitrary, it follows that 



limsupP sup ||1/"(t)|| >8A;e <limsupP((0"'^)'^). 

" \te[eo,T] J n 

Combining this with (74) (assuming without loss that, for each k, is 
bounded), (75) and the fact that e,eo > and T are arbitrary, it follows 
that /q e~"'^K'^ ds ^ 0. Therefore (49) holds and this concludes the proof 
that both SCPs satisfy the conditions of Theorem 4(i). Parts (i) and (ii) of 
Theorem 2 follows. 

(iii) The N-SCP (ii). To prove this part, it suffices to show that, for each 
5, there is a locally Lipschitz h' such that the N-SCP (i) applied to h' gives 

(83) lim sup ^ / e-^^L{X'^,u'l)dt<V{x) + 5. 

n— »oo JO 

Recall from the proof of Theorem 3 that /i : M'^ — > S'^ is a function satisfying 
Lp{x^h{x))= inf ip{x^v) =: ip* {x), 
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where 

(p{x, u) = b{x, u) ■ Df{x) + L{x, u). 
For each e > 0, let /i^ : M'^ ^ S'^ be a function defined as 

,er ^ T.y , V , e)h{y) 

h^x) = " — , 

Y.yd[x,y,e) 

where both sums extend over y € el}^ H eA;^/^), and d{x,y,e) denotes 
the Euchdean distance from y to the boundary dB{x,ek^^'^). It is easy to 
check that is locally Lipschitz. Write d{x,y,e) = d{x,y,e)/ J2y' d{x,y',e). 
By assumption, u L{x,u) is convex, and since u>-^b{x,u) is affine, n i— > 
(p(x,u) is convex. Using Jensen's inequality, uniform continuity of {x,u) i— > 
(p{x,u) and of x i— > f*{x) on compacts, for each 6 > 0, there is e such that 

ip{x,h%x)) = ip(^x,'^d{x,y,e)h{y)^ 

<J2d{x,y,e)ip{x,h{y)) 
y 

<Y,dix,y,eMy,hiy)) + 5/2 
y 

= Y,d{x,y,e)^*{y) + 6/2 
y 

<ip*ix) + 5. 

Everywhere in the above display, the sum extends over y E eZ'^ n B{x, e/c^/^). 
A review of the proof of Theorems 2 and 4 shows that, upon applying N- 
SCP (i) with /i*", (83) holds. By taking an appropriate sequence hn = h^", it 
is then clear that N-SCP (ii) applied to /i„ admits the conclusion of Theorem 
4, and therefore (32). □ 

5. Further research. 

5.1. Work- encouraging SCPs. We have restricted our analysis to work- 
conserving SCPs. However, our results regarding asymptotic optimality among 
all admissible SCPs hold, in fact, under the additional condition that the 
cost functions are work encouraging (cf. Definition 6). Recall that with each 
admissible SCP we have associated a cost of the form [cf. (20)] 

/■oo 

C7" = £;/ e-T*L($"(t),^"(t))dt 
Jo 
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Definition 6. We say that the cost function L (or the corresponding 
cost function L) is work encouraging if, for each n, the infimum of over all 
admissible SCPs is equal to that over all work-conserving admissible SCPs. 

Corollary 1. Let all assumptions of Theorem 2 apply, except the as- 
sumption that the SCPs are work conserving. Then the conclusions of The- 
orem 2 prevail, given that the cost function L is work encouraging. 

Although in many cases it is intuitively clear that work conservation is 
optimal (for P-SCPs, not for N-SCPs), in the presence of abandonments, 
and in the generality of our setting, this turns out to be nontrivial to prove. 
We intend to treat the issue in a future work. We end this section with 
a few examples that are intended to exhibit some of the subtleties of this 
point, and to indicate how it can be dealt with. The arguments should be 
considered as proof outlines only. 

First, consider the expected discounted number of customers of a par- 
ticular class, say class 1, present in the system. If 6i < Hi, then class-1 
customers leave the system faster when they are served than when they are 
in the queue. Hence a good policy will attempt to serve these customers as 
much as possible, and will be work conserving. On the other hand, if 9i> fii, 
then customers leave the system by abandoning the queue faster than by 
being served, and as a result, a policy which minimizes the cost will not 
schedule any services at all. 

More subtle are the costs associated with queue length and abandonment. 
We argue heuristically that if 9i> fii, then there are cost functions i, non- 
decreasing as a function of for each i, for which work conservation is 
actually not optimal. Suppose that 9i> ^i, and the cost is for $i < c, and 
1 for > c, where c > 0. If $i < c, no cost is incurred, and customers leave 
faster if in the queue than if in service. Thus an SCP that keeps customers 
in the queue would do better than a work-conserving SCP. 

Consider the case where I is linear in ^>j, and /Xj are arbitrary. We argue 
that work-conserving policies are optimal. We use coupling. A sample path is 
considered under an SCP that is not always work conserving. The coupling 
is used to show that if the SCP is changed to be work conserving, the cost 
will be no higher than for the original SCP. In view of the discussion on costs 
of abandonment, one can use the relation between abandonment rate and 
expected queue length to obtain the result. Consider a sample path under an 
SCP that leaves customers in the queue when there are idle servers. Modify 
it by moving a customer into service. Keep that customer in service until 
the earliest of: (i) it completes its service, (ii) its "twin" (i.e., the customer 
in the original system that is in the queue) abandons, or (iii) the original 
SCP needs to use the server. In cases (i) and (ii), the cost of the modified 
SCP will be no larger. In case (iii), it is the same as the original. In the case 
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where there is a class for which the abandonment rate is zero, the relation 
between abandonment and queue lengths cannot be used. However, this can 
be treated similarly to the following paragraph. 

Consider next the case where I is an increasing function of <l>j, for all i, 
with Qi < fii for all i. Here, when a customer is moved into service, its service 
time is coupled to the abandonment time of its twin: Pick an exponential 
random variable with rate //j, and a Bernoulli random variable that is 1 
with probability Oi/nt. The service time is the exponential random variable. 
If the Bernoulli random variable is 1 , then the abandonment occurs simulta- 
neously; otherwise the original customer does not abandon at that time and 
picks a new exponential random variable with rate 9i. Again, if the original 
SCP needs the server, the customer is moved out. It can be seen that the 
cost of the modified SCP will be no larger than the original one. 

5.2. Additional topics. The following is a list of research problems that 
are suggested by the present study. 

1. Nonlinear waiting costs: Nonlinear waiting costs are natural for quanti- 
fying human costs of waiting [37, 39]. We believe that it is possible to 
reduce such costs to nonlinear costs of queue lengths, and are planning 
to include this in future work. 

2. Alternative cost structures: Discounted costs are mathematically conve- 
nient. Long-run average costs provide an alternative which is no less, 
perhaps more, natural for call center applications. Their analysis, how- 
ever, would be mathematically more taxing. 

3. Performance analysis in the QED regime: In the present study, we are 
not analyzing the performance of our queueing system under the pro- 
posed SCPs. In particular, one would like to confirm that the (discounted) 
probability of delay, for each class, is nontrivial, as expected in the QED 
regime. Such analysis might require numerical supplements, as in [21]. 
This could also shed further light on qualitative features of our asymp- 
totically optimal SCPs. 

4. More general models: The model in Figure 1 is a beginning. Ultimately, 
one would like to generalize it to the model surveyed in [38], which has 
heterogeneous pools of servers with overlapping service skills. (See [14] 
for interesting simulations of such models.) In conventional heavy traffic 
(efficiency driven) , a simple generalized C control was proved asymptoti- 
cally optimal [30]. Here, only the problem of assigning servers who become 
idle is relevant, since customers essentially never encounter an idle server 
upon arrival. This same simplifying feature applies for our model, under 
work conservation. But with heterogeneous pools of servers, and with a 
nontrivial fraction of arrivals encountering idle servers (as expected in 
the QED regime), both the assignment of servers to customers and the 
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routing of arriving customers to idle servers become significant. In a call 
center context, the problem of online matching customers and servers is 
called skills-based routing; it is widely acknowledged as the most impor- 
tant and difficult operational problem next to staffing, to which we now 
turn. 

5. Staffing insights: The staffing problem is to determine the least (optimal) 
number of servers n that is required to conform to given performance 
standards. In the QED regime, n ~ i2 + (3\fR, where R is the offered load 
and /3 is a scalar. The problem can thus be decomposed, as in [7], into 
two steps: first, given a QED operation, determine the least (optimal) 
scalar /3; then, establish that operating in the QED regime is indeed 
desirable (optimal). The staffing problem becomes more interesting and 
far more difficult in a skills-based routing environment. ([8] is the single 
paper on the subject that we are aware of.) 

APPENDIX 

Proof of Proposition 2. Note that (x,n) i-^ b{x,u) is continuous and 
X I— > b{x,u) is Lipschitz uniformly in u. Consider bm, a function that agrees 
with b on the ball B{0,m), uniformly Lipschitz and bounded. Then strong 
existence and uniqueness for 

X^{t)=x + rW{t)+ [ bm.{Xn,{s),u{s))ds, 0<t<oo, 

JO 

holds by Theorem I.l.l of [6]. Since \\Xm{t)\\ < ||x|| +c||W^(t)|| + c/o ||X^(s)|| ds, 
one has ||Xm(t)|| < (||x|| + c||VF||j )(1 + e*^*) by Gronwah's lemma. Thus let- 
ting Tm = inf{t: ||Xm(t)|| > m}, one has ^ oo a.s. Therefore X{t) = 
lim.mXm{t) for all t defines a process that solves the equation (a strong 
solution). If X and X are both strong solutions, then, for every m, they 
both agree with X^ on [0,rm]- Therefore they agree on [0,cx)) a.s. □ 

Proof of Lemma 1. Let n G N and i £ K he fixed, and consider for 
each s the a-fields 

Fs = fT{l^^„^^^^^^, i?r(«) : n G M+, a < 4, 
a = a{RnP + 7) - RnP) : /3 > 7 > 0}. 

o 

We simplify notation by writing Tu =T^ {u) and R{u) = Rf{u). 

For each t and s, one has {Tt < s} E Fg, and therefore, for each t, Tt is a 
stopping time on the filtration {Fg). We next show that Mt := R{t) — Oit is a 
martingale on the filtration {Fg)] hence the lemma follows from the optional 
stopping theorem. Indeed, it is clear that Mg is measurable on Fg for each 
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s. Moreover, Mr — Mg is measurable on Gs for each s and r > s. It remains 
to show that Fs is independent of Gs for each s. Fix s. Fix 6, and n > 0, 
< a < s < s + _fC5 = ;3, 7 > 0. Let 

Note that P(Umi?m) = 1- Let 

HK,r = {Tr>s; f<K on [0, r]}. 

Let also 

Hm,K = {f<K, on [m6, (m + 1)5]}. 
For measurable bounded f,g (denote by c a bound on fg), 
Cf,g := E[f{l{T^^s},R{oc))g{R{fi + 7) - Rm] 

[r/8] 

= E E[lH^nH^JiMTu<s},R{(y))9{R{P + 7) - Rim + ei, 

where |ei| < cP{H'^^^). Under the event Hm n Hm,K, T^s < s < T(^m+i)S - 
s + K6 = p. Denote = (3 — T(^m+i)5^ and note that < Am < K5 under 
the same event. Then 

[r/S] 
m=0 

- R{T(m+l)5 + Am))} + ei. 

Let Hk,5 denote the event that there are no jumps of the process R within 
[s, s + U [s + 7, s + 7 + K5\: 

m=0 

+ 61+62 
[r/5] 

m=0 

+ 61+62+63, 

where |62|, I63I < cP{H1^^^). Recall that by Definition 2(i), J^"(t) and ^"(t) 

are independent. Since under i?m,, a < s < T'(m+i)5, it follows that 1^ ^^^^ /(1{T, 

i?(a)) G + 1)5). Also, R{T^^m+l)S + l) - R{T^m+i)s) G a"((?n + l^^). 
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and, using Definition 2(ii), it has the same law as Rij)- Hence 

[r/S] 

Cf,g = Eg{R{^)) J2 E{lH„.nH„,,JiMTu<s},Rim + ei + + 63 

m=0 

00 

= Eg{Rij)) ^ E{lH,J{l{T^<s},Ria))} + 61 + 62 + 63 + 64 

m=0 

= Eg{R{j))E{f{l{T^<,},R{a))} + ei + 62 + 63 + 64, 

where |e4| < cP{Hj^.^). It follows from Lemma 2 that £'||<I>"'||* < 00. Since 
T = <I>^, one has that 

lim liminf Pf^!<-r) = 1. 

Note also that P{H'^ ^) < ciK5 for some constant ci. Taking 6^0 and 
— > 00 such that K5 — > 0, and then taking r — > 00, we conclude that 

E[fil{T^^,y,R{a))g{R{s + 7) - R{s))] 
= E[f{l^T^^,y,R{a))]E[g{Ri^))] 
= E[fmT^^,},R{a))]E[giRis + 7) - R{s))]. 

Since a, s and 7 are arbitrary (subject to < a < s < s + ^), and so are f,g, 
it follows that Fs and Gg are independent for any s. The result follows. □ 

Proof of Proposition 1 (sketch). 

Existence and uniqueness for the system (7) and (11) are easily obtained 
by induction on the jump times of the processes A^, Rf and Sf-. By the 
assumptions on the function F, the constraints (6) are met. We next need 
to show that Definition 2 holds. For part (i) of the definition it suffices to 
show that, for any bounded measurable g, 

EigiAnrnt) +u)- AUrnt)), S^Tnt) + u) - S:{T^{t)), 

(84) i?r( fi^ (t) +u)- Rn m i g ^)i^] 

= Eg{Anu),Snu),Rnuy,i^K) 
where u> 0, and for part (ii) is suffices to show that 

E[g{SnTr{t)+uj)-SnTr{t)), 

(85) Rn T^" (t) + n,) - R^i f^" (t)); i e K, j > 1)1^^] 

= Eg{Snu,),Rnujy, ieK,j>l), 
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where < ui < n2 < • • • . In what fohows we suppress n from the notation, 
fix i and t and, denoting = TJ''{t), show that, for u> v > 0, 

(86) E[g{S,{Ti + n) - SiiT} + = Eg{Si{u - v)), 

(87) E[g{MTi{t) + n) - Ai(Ti(t) + ^;))|^i] = Eg{Ai{u - v)). 

Since the notation is quite comphcated, we do not give the fuh details on 
proving (84), (85), but only comment that the argument is similar to the 
one we use in proving (86) and (87). 

To show (86), for 6>0, let Hm = {T/ G [m6, (m + 1)6)} and 

Hm = {Si has no jumps on [m6 + v, {m + 1)6 + v]L) [m6 + u, {m + 1)6 + v\}. 

Then 

E[g{Si{Ti + u)-S,{Tl + v))\rt] 

oo 

= ^ E[lH.^g{S.,{Tl + u)- S,(Ti + v))\J't] 

m=0 

oo 



J2 E[lH^g{Si{im + 1)6 + u)- 5,((m + 1)6 + ^;))|.;^t] + ei 

71=0 

OO 

^ lH,„^[5(5^(("i + 1)6 + u)- Siiim + 1)5 + v))\Tt] + ei. 



m=0 



where 



|ei|<c^P(if™ni7^|.Ft). 

m=0 

Note that on the event T/ < rj, the quantities (X(s), ^(s), <l>(,s); s < t) only 
depend on A, R, Sj, j / i, and Si{t'), t' < rj. Since Si is Poisson and inde- 
pendent of the processes A, R and Sj, j i, using the definition of Tt and 
Hm we obtain that 

E[g{S,{Ti + u)-S,{Ti))\J^t] 



oo 



= ^ li^„^5(5i((m + 1)6 + u) - 5i((m + 1)6)) + ei 

m=0 

= ^g(5i(n))+ei. 
By (88), and since Hm depends only on Si{s); s > m6 + v, 

oo 

\ei\<cJ2lH^P{H'm)<c6, 

m=0 
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where c does not depend on 5 £ (0, 1). As a result, (86) holds. An equivalent 
of (86) for the processes Ri is proved analogously. Equality (87) is proved 
analogously, where one conditions on J^t V<T{rj(t)} and uses the fact that Ai 
is a renewal process. □ 

Proof of Proposition 3. Throughout, fix a compact subset A of X, 
and let c denote a positive constant that depends only on A, and whose value 
may change from location to location. Recall from the proof of Theorem 3 
that, for each x ^M.^, h{x) satisfies ip{x,h{x)) = inf^ggt ip{x,v), where 

<f{x, u) = b{x, u) ■ Df{x) + L{x, u). 

In the special case we analyze here, L{x,u) =J2i9i{i'^ •x)~^Ui), hence [cf. 
(26)] 

<^{x, u)= {£+{fi- e){l ■ x)+u - fix) ■ Df{x) + ^ 5i((l • x)^Ui) 

i 

=:a{x) + b{x) ■u + ^gi{xui), 

i 

where x = 1 • a; > 0. For any x G X, the map u ^ ^{x, u) is strictly convex; 
hence the infimum over S'^ is uniquely attained. 

Fixing x £ A, and letting mi{ui) = bi{x)ui + gi{xui), (p{x,u) is given as 
o(x) + J2i''^i{ui)- Use Taylor's formula for each rrii based at Uj, 

ip{x, v) = ip{x, u) + Y^ mi{ui){vi - Ui) + {l/2)m'l{ii){vi - Uif. 

i 

We claim that ^im[{ui){vi — Wj) > for v G S^. For if this is false, let 
u G S'^ be such that '^'i{ui){vi — Ui) = — c < 0. Then for :=u + £{v — u), 
J2i f^'i^iji^i —Ui) = —ce. Moreover, by assumption on the functions gi, there 
is a constant c such that g'/{xvi) < c; hence m'-{vi) < x'^c, v G S'^. Therefore 
J2i^i{Ci)i''^i ~ '^i)'^ ^ ce^, implying that ip{x,v'^) < ip{x,u) for e > small, 
contradicting the definition of u. 

Using the above, and that (^i) = x'^g'/{x^i) > cqxP' > c > on ^, we 
obtain 

ip{x,v) - Lp{x,u) > (1/2) Vm-'(^i)(ui -Ui)2 
(89) i 

>c\\v-u\\^, X e A, u = h{x), V eS''. 

Let x,y £ A and let u = h{x) and v = h{y). Since / is of class (cf. The- 
orem 3), 

||Z)/(x)-Z)/(y)||<c||x-y||. 
By the proof of Theorem 3, 

\H{x,p) - H{y, q)\ < c{\\p - q\\ + \\x - y\\^), 
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for p,q in a compact set. It follows that 

\^{x,u) - ^iy,v)\ = \H{x,DV{x))-H{y,DV{y))\ 
< c\\x — y\\^. 

Since by Assumption 2(iii) on L, x h-> ip(x,v) is Holder of exponent g, 

ip{x,v) — ip{x,u) < c\\x — y\\^. 

Combining the last display with (89), ||u — u|p = \\h{x) — h{y)\\'^ < c||x — 
and the result follows. □ 
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